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Overview 



High school dropouts face an uphill battle in a labor market that increasingly rewards skills and 
postsecondary credentials: they are more likely than their peers to need public assistance, be arrested 
or incarcerated, and less likely to many. This report presents results from a rigorous evaluation of 
the National Guard Youth ChalleNGe Program, an intensive residential program that aims to 
“reclaim the lives of at-risk youth” who have dropped out. More than 100,000 young people have 
completed the program since it was launched in the early 1990s. MDRC is conducting the evaluation 
in collaboration with the MacArthur Foundation Research Network on Transitions to Adulthood. 

The 17-month ChalleNGe program is divided into three phases: Pre-ChalleNGe, a two-week 
orientation and assessment period; a 20-week Residential Phase; and a one-year Postresidential 
Phase featuring a mentoring program. During the first two phases, participants live at the program 
site, often on a military base. The environment is “quasi-military,” though there are no requirements 
for military service. 

The evaluation uses a random assignment design. Because there were more qualified applicants than 
slots, a lottery-like process was used to decide which applicants were admitted to the program. 
Those who were admitted (the program group) are being compared over time with those who were 
not admitted (the control group); any significant differences that emerge between the groups can be 
attributed to ChalleNGe. About 3,000 young people entered the study in 10 ChalleNGe programs in 
2005-2006. 

Results 

A comprehensive survey was administered to about 1,200 young people in the program and control 
groups an average of three years after they entered the study, when they were about 20 years old, on 
average. Key findings from the survey include: 

• Members of the program group were much more likely than those in the control 
group to have obtained a General Educational Development (GED) certificate or a 
high school diploma and to have earned college credits. 

• Members of the program group were more likely to be employed at the time of the 
survey, and they earned about 20 percent more than their control group counterparts 
in the year before the survey. 

• There were few statistically significant differences between groups on measures of 
crime, delinquency, health, or lifestyle outcomes. 

These results are impressive; few programs for dropouts have produced sustained positive 
impacts. And yet, both the survey and a series of in-depth telephone interviews with ChalleNGe 
graduates suggest that many young people straggled to maintain momentum after leaving the 
residential program and returning home, where they had relatively few supports and also faced 
unusually challenging labor market conditions. ChalleNGe may want to experiment with ways to 
bolster its postresidential services to provide more support during this difficult transition. 
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Preface 



In the current labor market, where postsecondary education is prized, a high school dip- 
loma is a basic requirement for most employers. Across the United States, approximately a 
quarter of public high school students do not complete high school in four years. In urban areas, 
the number of students who drop out is significantly higher. While many of those who drop out 
eventually graduate or, more often, earn a General Educational Development certificate (GED), 
a long delay may place them at a serious disadvantage in competing for jobs and going on to 
college. In addition, a significant number of these young people become profoundly discon- 
nected from school, work, and society. 

Finding ways to reengage high school dropouts and help them move forward in educa- 
tion and the labor market is a pressing social issue. Over the past few decades a series of 
“second chance” programs, including the National Guard Youth ChalleNGe program, have 
targeted dropouts to help them finish high school (or get a GED) and get a foothold in the labor 
market. Created in the early 1990s, ChalleNGe has served more than 100,000 young people 
through a program model that includes an unusual and promising mix of features: an intensive 
Residential Phase with military-style discipline and a focus on promoting positive youth 
development, and a Postresidential Phase built around mentoring. 

MDRC, along with scholars from the MacArthur Foundation Research Network on 
Transitions to Adulthood, designed and implemented a random assignment evaluation of 
ChalleNGe in 10 sites. This report presents data on the program’s impacts, based mainly on a 
survey administered to the young people in the evaluation an average of three years after they 
entered the study. In addition, the report discusses the findings from a series of in-depth tele- 
phone interviews with ChalleNGe graduates who had recently completed the three-year survey. 

The results after three years are impressive: The ChalleNGe program group was still 
doing better than their counterparts in the control group in educational attainment, employment, 
and earnings. And the ChalleNGe graduates who were interviewed reported that the program 
was successful in changing their attitudes and bolstering their self-confidence. However, the 
survey results also show that several of the program’s key impacts have diminished over time, 
and the interviews suggest that many of the program's graduates are finding it difficult to 
sustain the progress they had made in meeting their education and employment goals. The next 
step for ChalleNGe may be to experiment with ways to enhance the Postresidential Phase of the 
program to offer participants continuing support when they return home to their communities 
and stronger connections to colleges and vocational training and jobs programs. 

Gordon L. Berlin 
President 
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Executive Summary 



Over the past three decades, broad economic shifts have dramatically reduced the avail- 
ability of well-paying jobs for workers without postsecondary education. In this context, young 
people who drop out of high school face particularly long odds of success. 

This report presents three-year results from a rigorous evaluation of the National Guard 
Youth ChalleNGe Program, which aims to “reclaim the lives of at-risk youth” who have 
dropped out of high school and give them the skills and values to succeed as adults. ChalleNGe 
is an intensive residential program that currently operates in more than half the states. More than 
100,000 young people have completed the program since it was launched in the early 1990s. 
MDRC, a nonprofit, nonpartisan research organization, is conducting the evaluation in collabo- 
ration with the MacArthur Foundation Research Network on Transitions to Adulthood. Several 
private foundations and the U.S. Department of Defense are funding the evaluation. 1 This may 
be the final report in the study; it is unclear whether additional follow-up data will be collected. 



The ChalleNGe Program 

The ChalleNGe approach grew out of a project by the Center for Strategic and Interna- 
tional Studies (CSIS) in the late 1980s and early 1990s that sought to develop new approaches 
for out-of-school youth. Staff in the National Guard Bureau in the U.S. Department of Defense 
developed the specific program model. They had concluded that many existing programs for 
disadvantaged youth were “focused on the symptomatic behaviors without understanding and 
addressing the underlying causes” and “placed limited, if any, focus on the post-program 
phase.” Thus, they designed ChalleNGe to be: 

. . .an intervention, rather than a remedial program. We would deal with the 
symptoms and underlying causes in a construct that fully embraced a “whole 
person” change and readied the students for the post-program environment. 

We would arm them with the skills and experiences necessary to succeed and 
we would ensure there was “a way back” to mainstream society. 2 



'The study is funded by Bill and Melinda Gates Foundation, Charles Stewart Mott Foundation, The Edna 
McConnell Clark Foundation, The John D. and Catherine T. MacArthur Foundation, the MCJ Foundation, The 
Robert Wood Johnson Foundation, and the William and Flora Hewlett Foundation. 

“Daniel Donohue, Designing a “ChalleNGe-like” Program for High School Dropouts and Students Who 
Are Drifting Through School, Disengaged, and Repeating Grades. (Fairfax Station, VA: Donohue Associates, 
LLC„ 2008). 
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In 1993, Congress funded a 10-site pilot of ChalleNGe. Funding was made permanent 
in 1998, and today there are 34 ChalleNGe programs in 29 states and Puerto Rico. 

States operate ChalleNGe programs under a Master Cooperative Agreement with the 
National Guard Bureau. Most states operate a single “100-bed” ChalleNGe program, serving a 
total of about 200 participants per year in two class cycles. A few states operate multiple 
programs or larger programs. The funding level for ChalleNGe — about $14,000 per participant 
— has not changed since the early 1990s. The federal government currently pays up to 75 
percent of the cost of the state programs, and states pay the remaining 25 percent. 3 

Although there is considerable room to tailor the program model to local conditions, the 
basic structure of the ChalleNGe program is the same in all states. The program is open to 
young people between the ages of 16 and 18 who have dropped out of (or been expelled from) 
school, are unemployed, drug-free, and not heavily involved with the justice system. 4 The 
program is open to both males and females, though about 80 percent of the participants are 
male. There are no income-based eligibility criteria. 

The 17 -month program is divided into three phases: the Pre-ChalleNGe Phase (two 
weeks), the Residential Phase (20 weeks), and the Postresidential Phase (one year). During 
the first two phases (totaling 22 weeks), the participants live at the program site, often on a 
military base. 

The first phase, Pre-ChalleNGe, is a physically and psychologically demanding assess- 
ment and orientation period. Candidates are introduced to the program’s rules and expectations; 
learn military bearing, discipline, and teamwork; and begin physical fitness training. 

Candidates who complete Pre-ChalleNGe are formally enrolled in the program as 
“cadets” and move to the second phase. The curriculum for the 20-week Residential Phase is 
structured around eight core components that reflect current thinking about how to promote 
positive youth development: Leadership/Followership, Responsible Citizenship, Service to 
Community, Life-Coping Skills, Physical Fitness, Health and Hygiene, Job Skills, and Aca- 
demic Excellence. Cadets spend the largest share of each day in the education component. 
During the study period, most programs helped participants prepare for the General Education- 
al Development (GED) exam; a few of them were able to offer a high school diploma. 



During the study period, the federal/state funding ratio was 60/40. The ChalleNGe legislation was 
amended in late 2009, raising the maximum federal share of program costs to 75 percent. 

4 ln order to be eligible for ChalleNGe, candidates must be 16 to 18 years of age and enter the program 
before their nineteenth birthday; a high school dropout/expellee; a citizen or legal resident of the United States 
and a resident of the state in which the program is conducted; unemployed; not currently on parole or on 
probation for anything other than juvenile status offenses, not serving time or awaiting sentencing, not under 
indictment or charged, and not convicted of a felony or a capital offense; and drug-free. 
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The program environment is described as “quasi-military”: The cadets are divided into 
platoons and squads, live in barracks, have their hair cut short, wear uniforms, and are subject 
to military-style discipline. The daily schedule is highly structured with almost no “down 
time,” and the cadets are closely supervised by staff at all times. While ChalleNGe uses 
military structure, discipline, facilities, and staff to accomplish its objectives, participation in 
the program is voluntary, and there are no requirements for military service during the program 
or afterward. 

Toward the end of the Residential Phase, the cadets work with staff to arrange a 
post-residential “placement.” Acceptable placements include employment, education, and 
military service. The cadets who successfully complete the Residential Phase move into the 
one-year Postresidential Phase, which involves a structured mentoring program. The Chal- 
leNGe mentoring program is unusual, in that young people nominate their own mentors during 
the application process. ChalleNGe initiates the mentoring relationship partway through the 
Residential Phase, after the staff screen and train the mentors. The staff then maintain contact 
with both the program’s graduates and their mentors at least monthly during the Postresidential 
Phase, working with and through mentors to help solve problems and to monitor the youths’ 
progress. 



The ChalleNGe Evaluation 

The National Guard Bureau collects extensive data on program participation and partic- 
ipants’ outcomes. However, for some time, officials in the Department of Defense and many 
ChalleNGe program directors have been eager to obtain more rigorous data on what difference 
the program makes. The National Guard Bureau’s outcome data do not address this question 
because there is no way to know to what extent the outcomes that program participants or 
graduates achieve are attributable to their participation in ChalleNGe; the program serves 
relatively motivated young people who were determined to make a change and might make 
progress without ChalleNGe. Thus, in 2004, the officials and directors began working with 
MDRC and the MacArthur Foundation Research Network on Transitions to Adulthood to 
explore the possibility of conducting a random assignment evaluation of the program. Ultimate- 
ly, the Department of Defense agreed to fund 20 percent of the first phase of the evaluation, and 
MDRC raised the remaining 80 percent from private foundations. 

In 2005, 12 state ChalleNGe programs agreed to participate in the evaluation. These 
programs were not chosen randomly. Rather, there was an effort to identify programs that had 
stable staffing and that tended to receive more applicants than they could serve. 

The evaluation uses a random assignment research design in which a group of young 
people who applied to ChalleNGe and were invited to participate (the program group) is 
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being compared over time with a second group (the control group) who also applied to 
ChalleNGe and were deemed acceptable, but were not invited to participate. Random assign- 
ment was conducted only during class cycles in which there were substantially more appli- 
cants than program slots. 

Because the study’s participants were assigned to the program group or the control 
group through a random process, one can be confident that any significant differences that 
emerge between the groups over time — for example, differences in educational attainment or 
employment rates — are caused by the ChalleNGe program. These differences are described 
as impacts. 

Ultimately, random assignment was conducted for 18 class cycles across 10 programs 
in 2005 and 2006 (two of the sites that agreed to participate were unable to conduct random 
assignment because there were not substantially more qualified applicants than slots). About 
3,000 young people entered the study. 

In order to preserve the integrity of the random assignment design, the evaluation is fol- 
lowing all of the young people who were assigned to the two research groups, even though 
some of those in the program group did not complete the ChalleNGe program. Data from the 
program’s management information system show that about 83 percent of the program group 
started the program, 68 percent completed the Pre-ChalleNGe Phase and formally enrolled, and 
53 percent graduated (that is, completed the Residential Phase). The graduation rate among 
enrollees — 78 percent — is very close to the national rate for the same time period. 5 

A series of surveys has been administered to the program and control groups over time. 
The first survey, a very brief interview, was conducted about nine months after the young 
people entered the study — not long after the program group had completed the Residential 
Phase. The results from that survey, presented in the study’s first report, were quite promising. 6 
The second survey, conducted an average of 21 months after study entry, also yielded a variety 
of positive results. 7 

This report presents the results from the third survey, which was conducted about three 
years after participants entered the study, more than a year after the Postresidential Phase had 
ended. Just under 1,200 young people were interviewed, and the response rate was 78 percent. 
Most respondents were 20 years old when they were interviewed. The report also discusses the 

5 The graduation rate among enrollees is calculated by dividing the overall graduation rate for the program 
group (53 percent) by the proportion of the program group that enrolled (68 percent). 

6 Dan Bloom, Alissa Gardenhire-Crooks, and Conrad Mandsager, Reengaging High School Dropouts: 
Early Results from the National Guard Youth ChalleNGe Evaluation (New York: MDRC, 2009.) 

7 Megan Millenky, Dan Bloom, and Colleen Dillon, Making the Transition: Interim Results of the National 
Guard Youth ChalleNGe Evaluation (New York: MDRC, 2010. 
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findings from a series of in-depth telephone interviews with ChalleNGe graduates who had 
recently completed the three-year survey. 



Results from the Three- Year Survey 

• The program group was much more likely than the control group to 
have obtained a GED and to have earned college credits. 

The top panel of Table ES.l shows that almost 56 percent of the control group had 
earned a high school diploma or a GED three years after entering the study. This is broadly 
consistent with national data, which show that most young people who drop out of high school 
eventually earn a diploma or, more commonly, a GED. 8 However, the figure for the program 
group is almost 72 percent, indicating that ChalleNGe substantially accelerated participants’ 
educational attainment. The asterisks show that the difference between groups — about 16 
percentage points — is statistically significant, meaning that it is very unlikely to have arisen by 
chance. As expected given the program model, many more program group members had earned 
a GED than a high school diploma, and the program’s impact was concentrated on GED receipt 
(the difference between groups in high school diploma receipt is not statistically significant). 
Program group members were almost twice as likely to report that they had earned at least one 
college credit, though relatively few people in either group reported that they were currently in 
college when interviewed (third panel). The impacts on educational measures have diminished 
somewhat since the earlier survey waves. 

• Young people in the program group were more likely to be employed at 
the time of the survey, and they earned about 20 percent more than 
their control group counterparts in the year before the survey. 

The second panel of Table ES.l shows three measures of sample members’ employ- 
ment and earnings in the year before the survey, which was administered during a deep reces- 
sion that disproportionately affected younger workers. Program group members earned an 
average of $13,515 during the period (this figure includes zeroes for sample members who did 
not work), about $2,267 (20 percent) more than the control group average. This earnings 
difference is driven, in part, by the fact that the program group worked more steadily during the 
period (about eight months of work versus seven months for the control group, on average). The 
third panel shows that program group members were also more likely to be currently employed 
at the time of the survey interview. 



David Hurst, Dana Kelly, and Daniel Princiotta, Educational Attainment of High School Dropouts 8 
Years Later (Washington, DC: National Center for Education Statistics, 2004). 
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National Guard Youth ChalleNGe Program 
Table ES.l 

Impacts on Selected Outcomes from the 3- Year Survey 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 3 


Educational attainment (%) 


Earned high school diploma or GED certificate 15 


71.8 


55.5 


16.2 *** 


0.000 


HS diploma 


30.3 


26.6 


3.7 


0.162 


GED 


56.9 


34.5 


22.4 *** 


0.000 


Earned any college credit 


34.9 


18.8 


16.1 *** 


0.000 


Employment history 

In the past 12 months 


Employed (%) 


88.4 


84.5 


3.9 * 


0.051 


Earnings ($) 


13,515 


11,248 


2,266 *** 


0.003 


Number of months employed 


8.1 


7.2 


q g *** 


0.001 


Current status 

Currently enrolled in ( %) 


High school or GED prep classes 


7.3 


9.2 


-1.9 


0.243 


College courses 


11.4 


7.8 


3.6 ** 


0.042 


Job training 


4.6 


3.1 


1.6 


0.180 


Working (%) 


57.8 


50.7 


1 j ** 


0.015 


Average weekly eamings c ($) 


240 


210 


30 * 


0.086 


Currently enlisted in the military (%) 


8.5 


7.4 


1.1 


0.494 


Involved in any productive activity 11 (%) 


63.6 


59.0 


4.6 


0.111 


Has HS diploma or GED and is currently involved in any 


productive activity 11 (%) 


49.1 


37.8 


j 1 4 *** 


0.000 


Crime and delinquency (%) 

Since random assignment 


Arrested 


50.6 


51.4 


-0.8 


0.777 


Convicted 


27.6 


24.9 


2.8 


0.294 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors for all 
impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 2.766, 2.898, 2.625, 
and 2.910. 

b Because this measure includes some respondents who indicated that they earned both a high school diploma 
and GED certificate, the percentages reported for the separate measures do not sum to the percentages reported for 
this measure. 

c Weekly earnings averages include zeroes for respondents who were not employed. 

d This measure includes any employment, school or GED programs, vocational training, military activities, or 
any residential programs (not listed separately above). 
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Interestingly, despite the National Guard's sponsorship of the ChalleNGe program, 
there was no statistically significant difference between groups in the percentage of sample 
members who were enlisted in the military at the time of the survey. Similarly, the two groups 
were about equally likely to have enlisted at any point since entering the study (not shown in 
Table ES.l). This result was confirmed using Department of Defense military enlistment 
records for active duty personnel. 

Overall, program group members reported spending substantially fewer months during 
the follow-up period being idle, or “doing nothing” — that is, they were not working, in school, 
or in the military (not shown in Table ES.l). 

• There were few statistically significant differences between groups on 
measures of crime, delinquency, health, or lifestyle outcomes. A few of 
the measures showed unfavorable trends. 

The study’s earlier reports found favorable impacts on several measures of criminal jus- 
tice involvement and health. As shown in Table ES.l, the criminal justice impacts were no 
longer evident at the three-year point; about half of each group reported at least one arrest since 
entering the study. Similarly, about two-thirds of each group reported being in good or excellent 
health (not shown in the table). 

Program group members were more likely to report that they were living on their own 
at the time of the survey (the largest share of respondents in both groups were living with their 
parents). This may indicate greater progress toward achieving a key adult milestone. On the 
other hand, other data indicate some potentially unfavorable trends. For example, program 
group members were more likely to report that they were not using birth control (it is possible 
that some of them were actively trying to have children) and that they had tried illegal drugs 
other than marijuana (these outcomes are not shown in Table ES. 1). 



Findings from the In-Depth Interviews 

MDRC staff conducted in-depth telephone interviews with 24 program group members 
just after they completed the three-year survey. The young people who were interviewed are not 
representative of the full program group because the in-depth study targeted only sample 
members who had completed the program' s Residential Phase. 

The young people who were interviewed spoke fondly of the ChalleNGe program and 
described how their participation had resulted in profound, positive changes in their attitudes, 
expectations, and self-confidence. Nevertheless, many of them struggled to maintain this 
momentum during the years after they completed the program. Most of the young people did 
not have strong family support, and few spoke much about the ChalleNGe mentoring compo- 
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nent. Most had moved through a succession of low-wage jobs; several had started college but 
had difficulty gaining traction. Such experiences are not unusual for young people in their early 
twenties, but they point to the challenge of building on the initial success of an intensive 
program like ChalleNGe. 



Conclusions and Next Steps 

Three years after entering the study (and one and a half years after the end of the pro- 
gram’s Postresidential Phase), young people who had access to ChalleNGe were doing better 
than their counterparts in the control group in several respects. They were more likely to have a 
GED, more likely to have at least started college, and more likely to be working. This is a 
notable achievement, since most rigorous evaluations of programs for disadvantaged youth have 
found that early impacts faded over time. 9 The lack of impacts in some other areas, such as 
crime and delinquency, is somewhat surprising, but since the sample members were only 20 
years old, on average, when they responded to the three-year survey, it is too early to know how 
their transition to adulthood will progress. The advantages program group members have gained 
as a result of their participation in ChalleNGe may continue to erode as control group members 
catch up, or they could grow as program group members build on their education and work 
experience. It is not clear at this point whether the evaluation will be able to collect further 
follow-up data. 

The main goals of the ChalleNGe Residential Phase are to help participants obtain a 
secondary school credential and develop attitudes and behavior patterns that will help them 
succeed in the future as students, workers, and citizens. The evaluation’s qualitative and 
quantitative data suggest that this phase succeeds in this respect for many participants. Yet, the 
data also suggest that it is difficult for many young people to maintain momentum afterward in 
a society and labor market that offer few opportunities for young people who have limited 
family support and do not follow a linear pattern from high school to college. The survey data 
show that several of the program’s key impacts have diminished in size over time, and, in the 
in-depth interviews, some program graduates (like many people their age) appeared to be 
having difficulty gaining a firm foothold in college or the labor market. 

The designers of the ChalleNGe program anticipated this issue and added an innova- 
tive Postresidential Phase built around mentoring to try to ease participants’ transitions back 
home. However, the evaluation found that the Postresidential Phase was implemented 
unevenly across sites. Moreover, while mentors may provide vital emotional support to young 



9 Dan Bloom and Ron Haskins, “Helping High School Dropouts Improve Their Prospects.” Policy Brief, 
The Future of Children (Princeton, NJ: Princeton-Brookings, 2010.) 
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people, they may not have the practical expertise and connections to help them find good jobs 
and succeed in college. 

Thus, the evaluation results to date suggest that the program may want to experiment 
with ways to enhance the Postresidential Phase of the program to help young people negotiate 
the difficult transition from the highly structured and supportive residential program. Possibili- 
ties might include: 

• Stationing some postresidential staff in the areas where concentrations of 
former participants live, rather than at the program site; 

• Using financial incentives to promote ongoing connections between former 
participants and the program, or to encourage success in school and/or work; 

• Promoting a stronger alumni support network through a combination of so- 
cial media and face-to-face activities such as “booster weekends” at the pro- 
gram site; 

• Developing stronger connections between ChalleNGe and community col- 
leges or other postsecondary institutions in areas where graduates live; 

• Building a stronger vocational training component, either during or after the 
Residential Phase of the program. 

Ideally, such enhancements, and others designed by program managers, would be rig- 
orously evaluated to determine whether they increase the program’s long-term impacts. Chal- 
leNGe is an example of a public program that already achieves impressive results, but might be 
improved through a systematic program of innovation and experimentation aimed at securing 
and sustaining the gains measured through three years. 
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Introduction 



Although there are fewer high school dropouts in this country than a decade ago, young 
people who drop out face long odds of success in a labor market that increasingly places a prior- 
ity on education and skills. 1 Nationally, about a quarter of high school freshmen do not graduate 
in four years; in the 50 largest U.S. cities, the dropout rate is closer to 50 percent. 2 While many 
of those who drop out eventually return to school and graduate or, more often, earn a General 
Educational Development certificate (GED), a long delay may place them at a serious disadvan- 
tage in competing for jobs and going on to college. 3 Moreover, a significant number of young 
people become profoundly “disconnected” from both school and work. In 2007, before the cur- 
rent recession, only 60 percent of 16- to-24-year-old high school dropouts worked at all during 
the year. Almost one in 10 young male dropouts were incarcerated. 4 

This report presents three-year results from an evaluation of the National Guard Youth 
ChalleNGe Program. ChalleNGe is an intensive residential program designed to “reclaim the 
lives of at-risk youth” who have dropped out of high school and give them the skills and values 
to succeed as adults. 5 The program currently operates in more than half the states, and about 
100,000 young people have completed the program since it was launched in the early 1990s. 
MDRC, a nonprofit, nonpartisan research organization, conducted the evaluation in collabora- 
tion with the MacArthur Foundation Research Network on Transitions to Adulthood. 6 



Earlier Evaluations of Youth Programs and the 
Origins of ChalleNGe 

During the last three decades, a number of rigorous evaluations have assessed programs 
that target disadvantaged youth. Some studies tested programs that primarily served youth who 



'U.S. Department of Education (2010). 

2 Chapman, Laird, and KewalRamani (2010). However, Roy and Mishel (2008) argue that graduation rates 
may be higher than reported in many recent studies. 

3 One national study tracked students who were in the eighth grade in 1988. About 20 percent of the stu- 
dents dropped out of high school at least once. Among the dropouts, 63 percent earned a high school diploma 
(19 percent) or a GED (44 percent) by 2000, eight years after their scheduled graduation date (Hurst, Kelly, 
and Princiotta, 2004). 

4 Sum, Khatiwanda, and McLaughlin (2009). 

5 Adapted from the National Guard Youth ChalleNGe Program’s mission statement. 

6 The evaluation is funded by Bill & Melinda Gates Foundation, Charles Stewart Mott Foundation, The 
Edna McConnell Clark Foundation, The John D. and Catherine T. MacArthur Foundation, The MCJ Founda- 
tion, The Robert Wood Johnson Foundation, The William and Flora Hewlett Foundation, and the U.S. De- 
partment of Defense. 



1 




were at risk but were still in the regular K-12 education system, while others studied “second- 
chance” programs for out-of-school youth, typically high school dropouts. 

The overall record from the studies of second-chance programs is mixed. On the one 
hand, several programs significantly increased the percentage of young people who earned a 
GED or another credential. In addition, some of the programs — particularly those that offered 
participants subsidized jobs — generated significant increases in employment or earnings in the 
short term. Others led to decreases in arrests or in other measures of criminal justice involve- 
ment. On the other hand, however, none of the studies that followed participants for more than a 
couple of years found lasting improvements in economic outcomes. Some of the studies did not 
report or collect long-term data, while, in other cases, early gains in earnings faded over time. 7 

In response to disappointing evaluation results in the 1980s, experts argued that youth 
programs should not just address problems or “deficits,” but rather should promote “positive 
youth development.” Specifically, they recommended that programs should go beyond educa- 
tion and training to expose young people to activities, settings, and relationships that are thought 
to promote healthy development. ChalleNGe, along with programs such as YouthBuild, is part 
of this movement. 8 

The ChalleNGe model grew out of a project by the Center for Strategic and Internation- 
al Studies (CSIS) in the late 1980s and early 1990s that sought to develop new approaches for 
out-of-school youth. The project’s final report concluded that aspects of the military structure 
could be beneficial for disadvantaged youth. 9 Many others have made this argument as well; for 
example, a report by the Brookings Institution concluded that “the United States military enjoys 
a well-deserved reputation for its ability to reach, teach, and develop young people who are rud- 
derless, and for setting the pace among American institutions in advancing minorities.” 10 

The CSIS report also concluded that the National Guard, with its strong community 
service mission, was well suited to operate a program for at-risk young people. Staff in the 
National Guard Bureau in the U.S. Department of Defense developed the program model for 
ChalleNGe. They had concluded that many existing programs for disadvantaged youth were 
“focused on the symptomatic behaviors without understanding and addressing the underlying 
causes” and “placed limited, if any, focus on the post-program phase.” Thus, they designed 
ChalleNGe to be: 



7 Bloom (2010). 

s YouthBuild programs serve youth ages 16 to 24. Participants work toward their GED or high school dip- 
loma while learning skills by building affordable housing. For more information, see www.youthbuild.org. 

9 Cullinan, Eaves, McCurdy, and McCain (1992). 

"’Price (2007). 
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...an intervention, rather than a remedial program. We would deal with the 
symptoms and underlying causes in a construct that fully embraced a “whole 
person” change and readied the students for the post-program environment. 

We would arm them with the skills and experiences necessary to succeed and 
we would ensure there was “a way back” to mainstream society. 11 

In 1993, Congress funded a 10-site pilot of ChalleNGe. Funding was made permanent 
in 1998, and today there are 34 ChalleNGe programs in 29 states and Puerto Rico. 



The ChalleNGe Model 

States operate ChalleNGe programs under a Master Cooperative Agreement with the 
National Guard Bureau. Most states operate a single “100-bed” ChalleNGe program, serving a 
total of about 200 participants per year in two class cycles (starting in January and July). A few 
states operate multiple programs or larger programs. 12 The funding level for ChalleNGe — 
about $14,000 per participant — has not changed since the early 1990s (in real terms, per- 
participant funding has fallen by about a th ir d during that time). During the study period, the 
federal government paid 60 percent of the cost of the state programs and states paid the remain- 
ing 40 percent. 13 

Although there is considerable room to tailor the program model to local conditions, the 
basic structure of the ChalleNGe program is the same in all states. The program is open to 
young people between the ages of 16 and 18 who have dropped out of (or been expelled from) 
school, are unemployed, drug-free, and not heavily involved with the justice system. 14 The pro- 
gram is open to both males and females, though about 80 percent of the participants are male. 
There are no income-based eligibility criteria. 

As shown in Figure 1, the 17-month program is divided into three phases: the Pre- 
ChalleNGe Phase (two weeks), the Residential Phase (20 weeks), and the Postresidential Phase 
(one year). During the first two phases (totaling 22 weeks), the participants live at the program 
site, often on a military base. 



11 Donohue (2008). 

12 For example, during the study period, Georgia operated two 200-bed programs, Illinois operated a single 
400-bed program, and Mississippi operated a single 200-bed program. 

13 The ChalleNGe legislation was amended in 2009, raising the maximum share of federal funding to 75 
percent. 

14 In addition, to be eligible for ChalleNGe, an applicant must be a citizen or legal resident of the United 
States and a resident of the state in which the program is conducted. “Not heavily involved with the justice 
system” means an applicant is not currently on parole or on probation for anything other than juvenile status 
offenses, not serving time or awaiting sentencing, not under indictment or charged, and not convicted of a felo- 
ny or a capital offense. 
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National Guard Youth ChalleNGe Program 
Figure 1 

ChalleNGe Program Phases 




1 7.5 Months 



The first phase, Pre-ChalleNGe, is a physically and psychologically demanding assess- 
ment and orientation period. Candidates are introduced to the program’s rules and expectations; 
leam military bearing, discipline, and teamwork; and begin physical fitness training. 

Candidates who complete Pre-ChalleNGe are formally enrolled in the program as 
“cadets” and move to the second phase. The curriculum for the 20-week Residential Phase is 
structured around eight core components that reflect current thinking about how to promote pos- 
itive youth development: Leadership/Followership, Responsible Citizenship, Service to Com- 
munity, Life-Coping Skills, Physical Fitness, Health and Hygiene, Job Skills, and Academic 
Excellence. Cadets spend the largest share of each day in the education component. At the time 
of the study, most programs helped participants prepare for the GED exam, but a few of them 
offered a high school diploma. 

The structure of the residential part of the program is designed to minimize some of the 
potentially negative effects of placing many at-risk young people together in a program setting 
— sometimes referred to as “deviant peer influences” or “peer contagion.” 15 The program envi- 
ronment is described as “quasi-military”: The cadets are divided into platoons and squads, live 
in barracks, have their hair cut short, wear uniforms, and are subject to military-style discipline. 
The daily schedule is highly stmctured with almost no “down time,” and the cadets are closely 
supervised by staff at all times. While ChalleNGe uses military structure, discipline, facilities, 



l5 Dodge, Dishion, and Lansford (2007). 
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and staff to accomplish its objectives, participation in the program is voluntary, and there are no 
requirements for military service during the program or afterward. 

Toward the end of the Residential Phase, the cadets work with staff to arrange a 
post-residential “placement.” Acceptable placements include employment, education, and 
military service. 

The cadets who successfully complete the Residential Phase move into the one-year 
Postresidential Phase, which involves a structured mentoring program. The ChalleNGe mentor- 
ing program is unusual, in that young people nominate their own mentors during the application 
process. ChalleNGe initiates the mentoring relationship partway through the Residential Phase, 
after the staff screen and train the mentors. The staff then maintain contact with both the pro- 
gram’s graduates and their mentors at least monthly during the Postresidential Phase to help 
solve problems and to report on the youths’ progress. 

The structured Postresidential Phase distinguishes ChalleNGe from many residential 
programs for youth. The purpose of this phase is to help ChalleNGe participants with the diffi- 
cult task of maintaining the new attitudes and behaviors they have learned in the Residential 
Phase when they return to their communities, families, and friends. 



The ChalleNGe Evaluation 

The ChalleNGe evaluation uses a random assignment research design in which a group 
of young people who applied to ChalleNGe and were invited to participate (the program group) 
is being compared over time with a second group (the control group) who also applied to Chal- 
leNGe and were deemed acceptable, but were not invited to participate. Because the study’s 
participants were assigned to one group or the other through a random process, one can be con- 
fident that any significant differences that emerge between the groups over time — for example, 
differences in educational attainment or employment rates — can be attributed to ChalleNGe. 
These differences are described as impacts. 

In 2005, 12 state ChalleNGe programs (almost half the state programs in existence at 
the time) agreed to participate in the evaluation. These programs were not chosen randomly. 
Rather, there was an effort to identify programs that had stable staffing and that tended to re- 
ceive more applicants than they could serve, a prerequisite for conducting a random assignment 
evaluation. See Table 1 for a list of the programs that agreed to join the study. 

The participating ChalleNGe programs recruited and screened applicants more or less 
as usual and identified a pool of applicants who met all eligibility criteria and were considered 
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National Guard Youth ChalleNGe Program 
Table 1 



Information on Participating Programs 



State 


Location 


First Year of 
Operation 


Number of Class Cycles During Which 
Random Assignment Was Conducted 3 


Arizona 


Queen Creek 


1993 


0 


California 


Camp San Luis Obispo 


1998 


1 


Florida 


Camp Blanding 


2001 


2 


Georgia 


Fort Gordon 


2000 


2 


Illinois 


Rantoul 


1993 


2 


Michigan 


Battle Creek 


1999 


3 


Mississippi 


Camp Shelby 


1994 


2 


New Mexico 


Roswell 


2001 


1 


North Carolina 


Salemburg 


1994 


2 


Texas 


Galveston 


1999 


2 


Virginia 


Camp Pendleton 


1994 


0 


Wisconsin 


Fort McCoy 


1998 


1 



SOURCE: National Guard Bureau (2005) and MDRC random assignment database. 



NOTES: Arizona and Virginia were unable to conduct random assignment because they did not have more 
qualified applicants than slots during the study period. 

a Most states serve about 100 participants during each of two cycles per year, which begin in January and 
July. Random assignment was conducted for a particular class cycle only if the number of acceptable 
applicants in the pool was at least 25 greater than the number of available program slots. 



acceptable. 16 Random assignment was conducted for a particular class cycle only if the number 
of acceptable applicants in the pool was at least 25 greater than the number of available program 
slots. 17 In other words, the programs would have had to turn away some applicants for these 
class cycles even without the evaluation. 18 To facilitate the evaluation, states agreed to use a 
random process to decide which qualified applicants to accept (at least one state already used a 



16 The Department of Defense authorized a modest amount of funding to support enhanced recruitment ef- 
forts by the programs that participated in the evaluation. 

17 Although the programs often refer to the number of available “beds,” in fact, the number of available 
slots is often determined not by physical space but by funding for staff. Typically, the programs are funded and 
staffed to graduate a certain number of participants per cycle (about 100 in most programs). During the study 
period, program managers told MDRC how many applicants they needed to accept in order to meet the gradua- 
tion target, assuming normal patterns of attrition. 

18 For the ChalleNGe evaluation, MDRC worked with the Department of Defense and the participating 
programs to develop a random assignment process that aimed to ensure that the evaluation would not reduce 
the number of young people who received ChalleNGe’ s services. MDRC’s Institutional Review Board re- 
viewed and approved the design. 
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random process but most did not). Also, to preserve the integrity of the design, applicants who 
were assigned to the control group were not allowed to reapply for later class cycles. 

Although 12 programs agreed to participate in the evaluation, there were many class 
cycles in which the number of applicants was too small to allow random assignment to take 
place. This occurred primarily because many programs tended to recruit only enough applicants 
to fill the available program slots. 

Therefore, as shown in Table 1, random assignment was conducted across only 10 of the 
12 programs. Arizona and Virginia were unable to conduct random assignment because they did 
not have more qualified applicants than slots during the study period. The total sample size 
(3,074) exceeded the original goal but was more heavily weighted toward the program group than 
originally intended (the sample includes 754 in the control group and 2,320 in the program group). 

Data Sources 

The evaluation draws data from several sources: 

• Baseline questionnaire. Applicants completed a two-page questionnaire that 
was inserted into the ChalleNGe application packet in the study sites. 19 These 
data provide a snapshot of the study participants just before they were ran- 
domly assigned to the ChalleNGe program or to the control group. 

• Program participation data. MDRC obtained information from the Chal- 
leNGe Data Management and Reporting System (DMARS), the national 
Web-based program tracking system used by all ChalleNGe programs. 

• Site visits. Members of the evaluation team conducted two-day visits to each 
of the 10 programs that conducted random assignment. Each visit included 
structured interviews with both program staff and participants. The study’s 
first report provided detailed information on the program’s implementation. 

• Follow-up surveys . 20 The first survey, a short questionnaire, was adminis- 
tered by phone or in person an average of nine months after members of the 
program and control groups entered the study. A more extensive survey was 
administered an average of 21 months after the young people entered the 
study. Results from these surveys were discussed in two earlier reports. A 



19 The applicants also signed a consent form to participate in the study at this point. If they were under age 
18 , a parent or guardian also signed the form. 

20 MDRC conducted a competition and selected Westat, Inc., to administer three follow-up surveys for 
the study. 
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third survey wave, approximately three years after sample members entered 
the study, was completed in December 2009. 

• Open-ended telephone interviews. MDRC research staff conducted inter- 
views by telephone with a small subsample of program group members 
who had both graduated from ChalleNGe and were interviewed in the third 
survey wave. 

The three-year survey, the source for results presented in this report, was administered 
from January to December 2009. It targeted the same 1,507 sample members as the previous 
survey (916 in the program group and 592 in the control group). 21 A total of 1,173 sample 
members (722 in the program group and 451 in the control group) completed the survey, for an 
overall response rate of 78 percent. The response rate was not significantly different across re- 
search groups. On average, when respondents completed the survey, they were 20 years old, 
and it had been 38 months since they had entered the study. 22 Overall, 88 percent of this sur- 
vey’ s respondents had been interviewed for the previous survey as well. 

A response bias analysis compared the baseline characteristics of those who completed 
the three-year survey with the full research sample. A separate analysis compared the fielded 
survey sample with those who completed the survey. Few significant differences emerged in 
each of these cases, and some differences are expected by chance. See Appendix A for more 
information on these analyses. 

Unless otherwise indicated, references to study participants for the remainder of this re- 
port refer to sample members who completed this three-year survey. 

Analysis Strategy 

To increase precision, the analysis of the survey data presented in this report controls for 
a number of baseline characteristics of the survey sample, including age, gender, race, whether 
the sample member was interested in ChalleNGe because he or she wanted to join the military, 
whether he or she lived in a two-parent household, and highest grade completed. Weighting was 
used in the full sample and subgroups to adjust for site size (so each of the 10 sites contributes 
equally to the results), survey response rates, and program versus control ratios. 23 



“'The sample was selected at differing rates across sites and random assignment status to minimize the va- 
riance of estimated impacts when sites are weighted equally in the analysis. 

22 Age of sample members at the time of survey ranged from 19 to 22. Survey completion ranged from 33 
to 52 months after random assignment. 

“'Alternate weights were created for the full sample that did not incorporate response rates, but these did 
not produce any differences in key outcomes. Impacts were also run without weights, so that sites with a larger 




Sample sizes in the individual sites are relatively small, so most of the analysis pools 
these weighted and regression-adjusted results from all the sites. Because study sites were not 
chosen randomly, technically the pooled results do not represent the overall impact of Chal- 
leNGe nationally. However, the study sites look like other ChalleNGe sites based on national 
performance data. 

In order to preserve the integrity of the random assignment research design, the impact 
analysis includes all program and control group members, including the program group mem- 
bers who did not participate in ChalleNGe or dropped out before completing the program. It 
also includes the small number of control group members who participated in ChalleNGe. 24 



Characteristics of the Study’s Participants 

Table 2 presents selected information from the two-page survey that sample members 
completed when they applied for ChalleNGe. These data provide a “snapshot” of the applicants 
as they entered the study. In general, the data indicate that ChalleNGe is serving a diverse group 
of high school dropouts. 25 In addition to the characteristics of the total sample, table columns 
present information on each of the research groups. The table shows a handful of statistically 
significant differences between the two groups, but one group does not appear more disadvan- 
taged or “at risk” than the other. For example, on average, those in the program group were 
more likely than the control group to have completed eleventh grade. However, they also re- 
ported higher rates of arrests, convictions, and drug or alcohol use. As mentioned above, the 
analysis controls for a number of these baseline characteristics. 

As shown in Table 2, most sample members were 17 years old at the point when they 
entered the study, and about 88 percent are male. 26 Roughly equal proportions described them- 
selves as white (42 percent) or African- American/black (34 percent); most of the rest described 



sample size account for a larger percentage of the total sample. Those unweighted impacts can be found in 
Appendix C. 

~ 4 A total of eight control group sample members enrolled in the ChalleNGe program, five of whom were 
in the three-year sample. 

25 Although ChalleNGe serves young people from 16 to 18 years old, the youngest applicants — those un- 
der age 16 and a half — were excluded from the evaluation; in other words, they were not subject to random 
assignment. Specifically, ChalleNGe applicants were excluded from random assignment if they would have 
been under 17 years old on the last day of the Residential Phase of the class cycle for which they applied. Ow- 
ing to this rule, the characteristics of the participants in the study do not necessarily match those of all the 
young people who participated in the programs during the cycles when random assignment occurred. 

26 In some cycles, it was not possible to include female applicants in the random assignment pool, because 
the programs needed to accept all or nearly all female applicants, as female staff had already been hired to 
work with them. Thus, the percentage of females in the research sample is slightly lower than the percentage of 
females in the programs. Typically, about 20 percent of graduates nationwide are female. 
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National Guard Youth ChalleNGe Program 
Table 2 



Selected Characteristics of ChalleNGe Survey Sample Members at the 
Time of Random Assignment 





Program 


Control 


Full 


Characteristic (%) 


Group 


Group 


Sample 


Gender 








Male 


87.6 


88.7 


88.0 


Female 


12.4 


11.3 


12.0 


Age in years 








16 


37.6 


35.5 


36.7 


17 


51.6 


55.8 


53.3 


18 


10.9 


OO 

oo 


10.0 


Race/ethnicity a 








Hispanic 


19.1 


16.6 


18.1 


White 


42.6 


42.0 


42.3 


Black 


32.4 


36.0 


33.8 


Other 


5.9 


5.4 


5.7 


Lives with 








Both biological parents 


26.0 


23.5 


25.0 


Mother only 


33.7 


37.6 


35.2 


Father only 


5.4 


8.2 


6.5 


One parent and a stepparent 


22.2 


21.7 


22.0 


No parental figures 


11.4 


7.9 


10.0 


Other combination 


1.3 


1.1 


1.2 


Anyone in household receives public assistance 


24.2 


29.6 


26.4 ** 


Highest grade completed 






** 


8th grade or lower 


12.5 


17.6 


14.5 


9th grade 


31.0 


28.3 


29.9 


10th grade 


36.6 


39.9 


37.9 


1 1th grade 


19.0 


14.0 


17.0 


12 th grade 


0.9 


0.3 


0.7 


Usual grades received in school 








Mostly As and Bs 


3.9 


3.6 


3.8 


Mostly Bs and Cs 


18.9 


16.3 


17.8 


Mostly Cs and Ds 


42.7 


40.0 


41.6 


Mostly Ds and Fs 


46.9 


46.8 


46.9 


Has/had Individual Education Plan (IEP) 


31.5 


31.2 


31.3 


Ever suspended from school 


83.7 


80.9 


82.6 


Ever arrested 


35.0 


26.6 


31.6 *** 


Ever convicted 


20.2 


14.7 


18.0 ** 


Sample size 


722 


451 


1,173 


(continued) 
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Table 2 (continued) 



Characteristic (%) 


Program 

Group 


Control 

Group 


Full 

Sample 


Who first suggested you should apply for ChalleNGe? 


Yourself 


24.5 


22.8 


23.8 


A relative 


47.4 


49.1 


48.1 


A school official 


17.1 


16.3 


16.8 


The justice system 


7.6 


7.5 


7.5 


Reasons for applying to ChalleNGe? 


Want a high school diploma/GED certificate 


82.2 


81.9 


82.1 


Want to go to college/get more training 


42.1 


45.8 


43.6 


Want to get a job 


39.0 


42.1 


40.2 


Want to join the military 


33.1 


34.5 


33.6 


Want to get life on track 


77.9 


82.0 


79.6 * 


Overall health very good or excellent 


67.0 


65.7 


66.5 


Taking any medication 


24.0 


21.5 


23.0 


Overweight (BMI 25-29) b 


20.7 


19.8 


20.3 


Obese (BMI 30+) b 


10.9 


11.4 


11.1 


Ever drink alcohol or use drugs 


39.7 


34.3 


37.5 * 


Sample size 


722 


451 


1,173 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 

NOTES: Missing values are not included in individual variable distributions. 

Distributions may not add to 100 percent because of rounding or where categories are not mutually 
exclusive. 

T-tests were used to assess differences in characteristics across research groups. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a The race/ethnicity categories shown here are mutually exclusive. For example, those listed as Hispanic 
includes sample members who may have indicated that they were Hispanic and black or white. None of these 
sample members are counted as black, white, or "other" in this table. 

b Body Mass Index (BMI) is a measure of body fat based on height and weight that applies to both adult 
men and women. BMI is calculated by dividing a person's weight by his or her height squared. A person is 
defined as overweight if his or her BMI is between 25 and 29.9. A person is defined as obese if his or her BMI 
is 30 or higher. 

themselves as Hispanic. Almost all are U.S. citizens and were bom in the United States, and 
only about 3 percent reported having any children of their own (not shown in the table). 

A quarter of the sample members lived with both biological parents when they entered 
the study; another 22 percent lived with a parent and a stepparent. More than 40 percent lived in 
a single-parent household (most commonly with their mother), and about 1 1 percent lived with 
no parent or stepparent. Less than 25 percent of sample members reported that their household 
received any public assistance, indicating that the ChalleNGe population is not, in general, ex- 
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tremely low income (though it is possible that some sample members were not aware that their 
household received public assistance). 

As expected, the participants in the study had not done well in school before leaving. 
Nearly half reported that their grades had been mostly Ds and Fs, and more than 80 percent re- 
ported that they had been suspended from school at least once. Nearly one-third reported that 
they currently or previously had an Individual Education Plan, which indicates special educa- 
tion status. 

About two-thirds of the study’s participants characterized their health as very good or 
excellent. On the other hand, about one-third were either obese or overweight. About one-third 
reported that they had used drugs or alcohol, though sample members may have underreported 
their drug use if they believed that the baseline survey was actually part of the program’s appli- 
cation process. 



Summary of Earlier Findings 

The evaluation’s earlier reports described the implementation of ChalleNGe in the 
study sites, discussed why young people had enrolled in the program, and presented results 
from surveys administered about nine and 21 months after sample members entered the study. 27 

Program Implementation and Participation in ChalleNGe 

Field research visits to all participating programs highlighted some variation across sites 
in the program environment, approaches to recruitment and discipline, and other elements of the 
program. Nevertheless, the basic structure of the program was quite similar from site to site, all 
of the programs were implementing all of the core elements of the program, and the staff were 
generally highly committed and professional. 

As shown in Table 3, about 83 percent of the young people who were assigned to the 
study’s program group actually started the program (that is, they showed up and registered); 
others may have changed their mind about participating after they were invited, or showed up to 
the program and failed a drug screen. Nearly 70 percent of the program group completed the 
Pre-ChalleNGe Phase and formally enrolled, and a little over half graduated from the Residen- 
tial Phase. The graduation rate among enrollees was about 78 percent, close to the national av- 
erage for this time period. 

An earlier report discussed potential challenges inherent in the structure of the mentor- 
ing component. Specifically, while mentoring can be critical to help ChalleNGe participants 



27 Bloom, Gaidenhire-Crooks, and Mandsager (2009); Millenky, Bloom, and Dillon (2010). 
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National Guard Youth ChalleNGe Program 
Table 3 



Program Participation and Completion Rates 
Program Group Members, Study Sample 



Outcome (%) 


Full 

Program Group 


Among 
Those Registered 


Among 
Those Enrolled 


Registered 


82.5 


100.0 


100.0 


Enrolled 


67.9 


82.3 


100.0 


Graduated 


52.7 


63.9 


77.7 


Sample size (total = 2,320) 


2,320 


1,913 


1,575 



SOURCE: MDRC calculations using data from the ChalleNGe Data Management and Reporting 
System (DMARS). 

NOTES: Registered participants include those who begin the Pre-ChalleNGe program, a 
physically and psychologically demanding assessment and orientation period. Enrolled 
participants include those who completed the Pre-ChalleNGe program, are officially enrolled in 
ChalleNGe as "cadets," and begin a 20-week Residential Phase. Graduated participants include 
those who successfully completed the Residential Phase and moved on to the 1-year 
Postresidential Phase. 



stay on track, the quality of the mentoring varied from site to site. In some cases, the cadets and 
families did not take the process seriously or did not have a strong pool of positive role models 
to choose from. All mentors were required to receive mentor training, but programs executed 
this training inconsistently; in some cases mentors received no more than a few hours of train- 
ing over the phone. Cadets were required to have a minimum of four contacts with their mentor 
per month during the Postresidential Phase, and mentors were asked to send in a monthly report 
on these contacts. However, in some programs, few resources were dedicated to monitoring or 
enforcing this, resulting in varying levels of adherence to the mentoring model. 

Results from the Nine-Month Survey 

The nine-month survey was quite brief, but the results were fairly striking. The program 
group was more than four times more likely than the control group to have a high school diplo- 
ma or a GED. The program group was also significantly more likely to be working and to be in 
college, and less likely to have been arrested since entering the study. There were also statisti- 
cally significant outcomes in reducing obesity and improving overall health and self-efficacy. 
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Despite these positive results, the report interpreted the findings cautiously. It pointed 
out that the survey had been conducted very early in the follow-up period — in fact, when most 
program group respondents were still participating in the Postresidential Phase. 

Results from the 21 -Month Survey 

A second report included promising results from a more comprehensive survey that was 
administered about 21 months after a sample member entered the study. At that time, the pro- 
gram group was still much more likely than the control group to have obtained a high school 
diploma or a GED and to have earned college credits. For example, about 61 percent of the pro- 
gram group had earned a diploma or a GED, compared with 36 percent of the control group. In 
general, program group members were somewhat more likely to be engaged in productive ac- 
tivities. For example, 72 percent of the program group was working, in school or training, or in 
the military, compared with 66 percent of the control group. 

Once again, these results were presented as promising but with reason for caution. Dif- 
ferences in educational attainment between the two groups shrank between the nine-month and 
21 -month points, and many control group members were attending high school or GED prep 
classes at the time of the interview. In addition, it was unclear how ChalleNGe participants 
would use their high school or GED credentials in the future. 

Roadmap 

The remainder of the report is organized as follows: The second section presents the 
impact findings from the three-year survey. The third section discusses the themes that emerged 
from in-depth interviews with a sample of ChalleNGe graduates. The last section discusses 
some conclusions and ideas for future research. 
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Three- Year Survey Results 



This section uses data from the three-year follow-up survey to examine the impacts of 
ChalleNGe on sample members’ education attainment and their current activities, including 
work, education, training, and military enlistment. As a reminder, at the time of this survey, the 
average age of sample members was 20. Two of the program’s key goals are to help young 
people further their education and get a foothold in the labor market. Beyond education and 
career exploration, the program was designed to foster a “whole person change” and help youth 
find “a way back” to society. Thus, the survey also examined a wide range of measures of 
health, crime and delinquency, psychosocial development, and civic engagement. 



Education and Training 

Overall, the survey data suggest that, as a result of their participation in ChalleNGe, the 
program group had better educational outcomes than the control group. Specifically, a greater 
proportion of the program group had earned a high school diploma or a General Educational 
Development certificate (GED), earned college credit, or received vocational training. 

• The program group was much more likely than the control group to 
have obtained a GED. 

As shown in Table 4, more than half the control group had received a high school dip- 
loma or a GED at the time of the survey. National data show that most high school 
dropouts eventually get a diploma or, more commonly, a GED. 28 However, ChalleNGe still had 
a large impact: 72 percent of the program group had earned a GED or a high school diploma, 
compared with 55 percent of the control group. This 16 percentage point impact was driven by 
significantly more program group members earning a GED. 

Interestingly, about 15 percent of the program group reported having both a high school 
diploma and a GED certificate. This is an unusual combination, since a GED is usually seen as 
an alternative to a high school diploma. Further analysis showed that the program group mem- 
bers who reported having both a diploma and a GED are concentrated in a few ChalleNGe sites. 
One of those is an alternative high school, and in another, an arrangement with the state legisla- 
ture allows young people who complete ChalleNGe and pass the GED exam to receive a state 
high school diploma. 

Receipt of a high school diploma, rather than a GED certificate, may be a key distinc- 
tion, since the value of a GED in the labor market is debatable. Studies find that GED holders 



2s Hurst, Kelly, and Princiotta (2004). 
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National Guard Youth ChalleNGe Program 
Table 4 



Impacts on Education and Training 



Outcome (%) 


Program 

Group 


Control 

Group 


Impact 


P- Value 3 


Earned high school diploma or GED certificate 


71.8 


55.5 


16.2 *** 


0.000 


HS diploma 


30.3 


26.6 


3.7 


0.162 


GED 


56.9 


34.5 


22 4 *** 


0.000 


Both HS diploma and GED 


15.2 


5.5 


9 2 *** 


0.000 


Earned any college credit 


34.9 


18.8 


16.1 *** 


0.000 


Received college degree 0 


0.9 


0.0 


0.8 * 


0.053 


Ever received vocational training 


40.1 


33.1 


7 Q ** 


0.016 


Received trade license/training certificate 

Current status 

Currently enrolled in 


29.7 


27.9 


1.9 


0.490 


High school or GED prep classes 


7.3 


9.2 


-1.9 


0.243 


College courses 


11.4 


7.8 


3.6 ** 


0.042 


Job training 


4.6 


3.1 


1.6 


0.180 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 

“Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors 
for all impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 2.766, 
2.898,1.859,2.625. 

b All degrees received are associate's degrees. 



earn significantly less than high school graduates. Studies have also shown that employment 
outcomes for those with a college degree are similar for GED holders and regular high school 
graduates, but that only a small minority of GED holders complete even one year of postsecon- 
dary education . 29 These data may help to explain why past evaluations of youth programs that 
substantially increased GED receipt did not find evidence of longer-term gains in employment 
or earnings . 30 



29 Tyler (2005). 

30 Bloom, Levy Thompson, and Ivry (2010). 
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• The program group was more likely than the control group to have re- 
ceived vocational training, earned college credit, or to be enrolled in col- 
lege at the time of the survey. 

Looking beyond high school completion, the program group was more likely to have 
received vocational training (40 percent of the program group compared with 33 percent of the 
control group). Some ChalleNGe programs offered training; however, among program group 
members who reported receiving some training, 80 percent reported that the training was out- 
side of ChalleNGe. Sample members received training certificates or trade licenses in a number 
of fields, including health care, computers, automotive, and construction. Health care and com- 
puters were the two most common areas for certification. Without further information about the 
level of training sample members received, it is difficult to assess how the training might affect 
labor market outcomes. 3 1 At the time of the interview, few study participants were still involved 
in a vocational training program. 

In addition to the increased training, Table 4 shows that the program group was nearly 
twice as likely as the control group to have completed at least one college course for credit. 
While college courses are available at some ChalleNGe sites, a very small percentage of the 
program group (3 percent) reported attending college courses as part of ChalleNGe (not shown 
in table). Therefore, the difference between the groups extends beyond program group mem- 
bers’ college course attendance at a ChalleNGe site. Overall, program group members earned 
more college credits than the control group (not shown in the table), and a small percentage of 
young people in the program group reported earning enough credits to receive an associate’s 
degree. 12 At the time of the interview, relatively few sample members (less than a quarter) were 
furthering their education or taking training courses; however, a higher percentage of the pro- 
gram group than the control group was enrolled in a college course. 



Employment and Military Enlistment 

Table 5 describes sample members' employment and military enlistment since random 
assignment. Current employment rates — about half the sample was employed — are very sim- 
ilar to the previous survey. The fact that the employment rate did not increase over time, even 
though respondents are getting older, is not surprising given the sharp rise in unemployment 



'The survey instrument asked one open-ended question about the type of trade license or training certifi- 
cate sample members received. Responses varied considerably; thus it is difficult to understand the value and 
seriousness of the training based on the responses. 

32 On average, program group members reported earning 3.5 college credits, and control group members 
earned about 1.8 college credits. These averages include all sample members; therefore, participants who had 
not earned any college credit have zero college credits in the calculations. 
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National Guard Youth ChalleNGe Program 
Table 5 

Impacts on Employment and Military Enlistment 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 3 


Employment history 

In the last 12 months 


Employed (%) 


88.4 


84.5 


3.9 * 


0.051 


Earnings ($) 


13,515 


11,248 


2,266 *** 


0.003 


Number of months employed 


8.1 


7.2 


q g *** 


0.001 


Current emnlovment 


Currently working 15 (%) 


57.8 


50.7 


7 j ** 


0.015 


Current hourly wage under $6 


8.2 


3.1 


g 2 *** 


0.000 


Current hourly wage between $6 and $7.99 


10.3 


13.6 


-3.2 * 


0.095 


Current hourly wage between $8 and $9.99 


15.2 


15.2 


0.0 


0.984 


Current hourly wage $10 or more 


23.1 


18.0 


5.1 ** 


0.036 


Current average weekly earnings 1- ($) 


240 


210 


30 * 


0.086 


Currently working full time d (%) 


50.1 


47.2 


2.9 


0.328 


Enlistment history ( % ) 


Ever enlisted 


18.3 


17.2 


1.1 


0.616 


Active Duty 


11.9 


10.9 


1.0 


0.573 


Reserves or National Guard 


6.3 


6.1 


0.2 


0.891 


Current enlistment ( % ) 


Currently enlisted in the military 


8.5 


7.4 


1.1 


0.494 


Active Duty 


4.8 


4.9 


-0.1 


0.929 


Reserves or National Guard 


3.7 


2.5 


1.2 


0.249 


Current activity 6 ( % ) 


Involved in any productive activity 


63.6 


59.0 


4.6 


0.111 


Has high school diploma or GED certificate and is 


currently involved in a productive activity 


49.1 


37.8 


4 *** 


0.000 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors for all 
impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 1.442, 2.910. 

b Due to missing wage information for some sample members, percentages in wage categories do not sum to 
total percentage currently working. 

c Weekly earnings averages include zeroes for respondents who were not employed. 

d Working full time refers to working at least 30 hours per week. 

'These measures include any employment, school, GED programs, or vocational training (as shown in Table 
4), military activities, or any residential programs (not listed above). 
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nationally that has accompanied the most recent recession. In 2009, when the survey was ad- 
ministered, the unemployment rate for youth reached 15 percent, the second-highest rate since 
the government began tracking these data. 33 

• The program group was significantly more likely to be employed and 
was earning more than the control group at the time of the survey. 

Even with the recession at its peak, program group members were more likely to be 
employed than control group members (58 percent compared with 51 percent). As the hourly 
wage categories in the middle of Table 5 illustrate, the program group was more likely than the 
control group to be in the highest and lowest wage categories. Eight percent of the program 
group and 3 percent of the control group reported wages under $6 per hour. This is often seen in 
programs that encourage employment, since people who might not have otherwise sought em- 
ployment are joining the workforce. In addition, in this case it appears that many sample mem- 
bers in this wage category were employed in jobs that included tips, such as waiting tables, 
where the hourly wage would not necessarily reflect their total earnings. 34 

Almost one-quarter of the program group reported an hourly wage of $10 or more, 
while less than 20 percent of the control group was receiving a similar wage, suggesting that 
ChalleNGe helped some people obtain higher-paying jobs. Further reinforcing these differences 
in employment and wages, average weekly earnings at the time of the survey were $240 for the 
program group and $210 for the control group. Weekly earnings take into account employment 
rates, total work hours, and hourly wages. 35 

• In the year before the survey, the program group was employed for 
more months and had higher earnings than the control group. 

Sample members reported on prior jobs as well, as shown in Table 5. Nearly the entire 
sample reported some type of employment since random assignment (not shown in the table). 
However, looking only at the previous 12 months, the program group showed greater employ- 
ment stability than the control group. The program group was employed for an average of eight 



33 The 15 percent unemployment rate is representative of young people ages 20 to 24 in the United States 
in 2009. In the same year, the unemployment rate for 18- and 19-year-olds was higher at 23 percent. (U.S. De- 
partment of Labor, 2010). 

34 The federal minimum wage during the time of the interviews ranged from S6. 55/hour to S7. 25/hour. For 
employment that includes tips, such as restaurant work, the minimum wage is lower. If respondents were work- 
ing in two jobs, the hourly wage data refer to the job in which they worked the most hours per week. 

35 The average weekly earnings calculation includes all current jobs reported by a sample member and in- 
cludes earnings of zero dollars for those not currently working. Less than 2 percent of the entire sample re- 
ported weekly earnings of more than $1,000, and the majority of those working repotted weekly earnings be- 
tween $200 and $750. An analysis of reported wages was conducted, removing the questionably high wages, 
and results were similar to those repotted in Table 5. 
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out of 12 months, while the control group was employed for an average of seven months in the 
past year. 

This difference in employment stability is also reflected in the earnings of the two 
groups. Over the past 12 months the program group reported earnings of approximately 
$13,500, about $2,300 higher than the control group average of $11,250. Given the differences 
in hourly wages and weekly earnings discussed above, one could hypothesize that these higher 
earnings over the past year are a reflection of higher wages, perhaps more hours worked each 
week, and higher rates of employment by the program group. 

• The program and control groups had similar rates of enlistment in the 
military. 

In view of ChalleNGe’s National Guard sponsorship and quasi-military environment, 
one might expect that a natural next step for many of the program’s participants would be to 
enlist in the military. In fact, more than a th ir d of study participants reported at baseline that 
they were interested in ChalleNGe in part because they wanted to join the military (see Table 2). 
In addition, as discussed earlier, enlistment is one of the three acceptable “placements” for ca- 
dets after they complete the ChalleNGe Residential Phase. 

As shown in Table 5, however, there are no differences in military enlistment — cur- 
rent or past — between the program and control groups three years after random assignment. 36 
Less than 20 percent of sample members reported enlisting since they entered the study, with 
more reporting active duty enlistment than Reserve or National Guard participation. 37 The ma- 
jority of enlisted sample members, at each time point, were either in the active Army or Army 
National Guard (not shown). To respond to concerns that enlisted sample members may have 
been harder to reach for the survey and thus possibly undercounted and underrepresented in this 
evaluation, a separate analysis of enlistment using military administrative data was conducted. 



36 Current enlistment rates are about 10 percentage points lower than reports by survey respondents of 
“ever” enlisting. The current rates are also lower than at the time of the 21-month survey, where the program 
group was significantly more likely than the control group to report that they had enlisted in the military (11 
percent versus 6 percent). There are a number of possible explanations for these differing rates — given that 
few enlistees would have completed their military service during this time. One reason may be participation in 
the Delayed Entry Program (DEP), which gives recruits up to a year prior to boot camp to change their mind 
about active duty military enlistment. Thus, some respondents may have reported that they enlisted when they 
entered the DEP, but then changed their mind and were no longer enlisted at the time of the survey. Another 
possible reason is that a sample member may have signed up to join the military but had not successfully com- 
pleted boot camp. In these cases and others, a person may have indicated they had enlisted in the past but were 
not currently enlisted at the time of the latest survey. 

37 Overwhelmingly, sample members who reported that they were currently enlisted in the military also re- 
potted that they were currently working. 
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The administrative data dispelled these concerns; these data showed enlistment rates that were 
similar to those in the survey data. 38 

The information in Tables 4 and 5 illustrate a variety of activities that sample members 
may be involved in as they make the transition to adulthood and become productive members of 
society. The bottom of Table 5 aggregates these activities. At the time of the interview, approx- 
imately 60 percent of the sample was working, furthering their education, increasing their job 
skills, or enlisted in the military. While these rates were similar across groups, the difference 
emerges in looking at those who had already received a high school diploma or a GED certifi- 
cate and were involved in another productive activity. ChalleNGe participants did better than 
the control group on this measure of progress; nearly 50 percent of the program group had a 
diploma or a GED and was either working, taking college courses, or enlisted in the military. 
Less than 40 percent of the control group was similarly credentialed and engaged. 

Figure 2 summarizes some of the key employment and education results from the 21- 
month (wave 2) and the three-year (wave 3) surveys for those who completed both surveys. 
Figure 3 illustrates the impacts (differences between the program and control group responses) 
at each wave. Asterisks in this figure indicate whether the differences between the two groups 
are statistically significant. The figures show that, overall, the trend of ChalleNGe’ s impacts is 
similar at the two time points, though some impacts declined over time. 

• The program group reported shorter periods of inactivity since random 
assignment than the control group. 

In an attempt to further understand how sample members had been spending their time, 
the survey asked about time “basically not doing anything” since random assignment. The sur- 
vey defined this inactivity (or “idleness”) as time when the respondent was not working, in 
school, in a training program, or in the military. As shown in Table 6, across the entire sample, 
more than two-thirds of the sample reported that there was one or more months during which 
they were basically “doing nothing” since entering the study. For this age group, some down- 
time is unsurprising. 

However, Table 6 shows that the control group spent significantly more months “doing 
nothing” during this important period of their development into young adults. Nearly 20 percent 
of the control group reported a year or more of inactivity, while just over 10 percent of the pro- 
gram group reported the same period of inactivity. These reports of idleness reinforce the find- 
ing that the program group had higher rates of participation in productive activities and educa- 
tional attainment than the control group. 



^However, the administrative data did illustrate that within the program group, those who graduated from 
ChalleNGe were more likely to enlist than those who did not complete the ChalleNGe program. 
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National Guard Youth ChalleNGe Program 
Figure 2 
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Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 











National Guard Youth ChalleNGe Program 
Figure 3 

Impacts on Selected Outcomes Among Respondents to Both the 
21-Month (Wave 2) and the 3-Year (Wave 3) Survey 
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diploma courses 

Impact 



■ Wave 2 
□ Wave 3 



SOURCE: MDRC calculations from responses to the 21 -month (wave 2) and the 3-year (wave 3) survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample members' 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 



Other Outcomes 

The ChalleNGe model was designed to go beyond education and employment goals to 
foster a “whole person change” and help youth find “a way back” to society. Thus, the survey 
also examined a wide range of measures of health, crime and delinquency, psychosocial devel- 
opment (life-coping skills), and civic engagement. Overall, there were no clear patterns of posi- 
tive impacts among these measures that were attributable to participation in ChalleNGe. 

• No consistent differences emerged between the two groups on delin- 
quency, life skills, or civic engagement outcomes. 

High school dropouts, particularly males, are at high risk of becoming involved with 
the criminal justice system. Moreover, about a third of study participants reported at least one 
arrest before they entered the study (see Table 2). By engaging young people and putting them 
on a more positive path, ChalleNGe hopes to reduce criminal activity in the future. The three- 
year survey asked sample members about arrests, convictions, sentences, and other delinquent 
incidents. 
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National Guard Youth ChalleNGe Program 
Table 6 



Impacts on Time Spent Idle Since Random Assignment 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 2 


Any time spent idle 


61.9 


72.7 


-10.7 *** 


0.000 


Idle for 1-2 months 


14.3 


10.8 


3.5 * 


0.082 


Idle for 3-11 months 


35.5 


42.4 


-6.9 ** 


0.018 


Idle for a year or more 


11.7 


19.2 


-7.5 *** 


0.000 


Sample size (total = 1,172) 


722 


450 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors 
for all impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 2.801, 
2.099. 



The top panel of Table 7 shows that half the sample reported at least one arrest, and a 
quarter reported a conviction since entering the study. Most convictions, which include all 
convictions since random assignment, were for public order or drug crimes. The self-reported 
delinquent incidents in the bottom panel of Table 7 reflect incidents in the past year and may 
include some incidents that are reflected in the top panel as well. The table breaks down the 
incidents as mainly property or violent ones. As Box 1 shows, violent incidents refer to any 
type of physical aggression; in this sample most incidents were fights that did not include wea- 
pons. Overall, no differences appeared between the two groups in the three-year survey on any 
of these measures. 39 

In addition to the outcomes discussed above, the survey asked a series of questions 
about civic engagement, leadership, and life-coping skills. Measures were intended to evaluate 
aspects of core program components or objectives such as responsible citizenship, service to 
the community, and group interaction abilities. These questions were selected or adapted from 
existing measures or were unique items developed and piloted by the MacArthur Research 



39 In the 21 -month survey, the program group was less likely to have been convicted of a crime than the 
control group in the year prior to the interview. However, the three-year survey asks about arrests and convic- 
tions back to the time sample members entered the study. Therefore, the information collected at the two time 
points is not directly comparable. 
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National Guard Youth ChalleNGe Program 
Table 7 



Impacts on Delinquency and Criminal Activity 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Self-reported arrests/convictions (%) 


Arrested 


50.6 


51.4 


-0.8 


0.777 


Charged with any crime 


37.1 


33.5 


3.6 


0.208 


Convicted of any crime a 


27.6 


24.9 


2.8 


0.294 


Convicted of a violent crime 


2.1 


2.3 


-0.3 


0.768 


Convicted of a property crime 


5.4 


5.6 


-0.3 


0.844 


Convicted of a drug crime 


8.1 


5.9 


2.2 


0.155 


Convicted of a public order crime 


11.2 


13.3 


-2.0 


0.296 


Self-reported delinauencv b 


Any incidents (%) 


57.1 


55.3 


1.8 


0.544 


Any violent incidents (%) 


48.7 


44.5 


4.2 


0.157 


Any property incidents (%) 


15.8 


18.1 


-2.3 


0.308 


Number of incidents 


1.5 


1.5 


0.0 


0.813 


Number of violent incidents 


0.9 


0.8 


0.1 


0.388 


Number of property incidents 


0.5 


0.5 


-0.1 


0.334 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

Results for arrests and convictions cover the time since random assingment. Self-reported delinquency 
results cover the 12 months prior to the survey interview. 

“Categories of conviction are based on definitions from Langan and Levin (2002). Categories may sum 
to more than the percentage convicted because a person may be convicted of more than one crime during 
the follow-up period. 

b See full list of types of delinquency in Box 1 . 



Network on Transitions to Adulthood. As discussed in detail in an earlier ChalleNGe report, 
these measures are still in development and exhibited little variation and few impacts at the 21- 
month survey. Few differences were seen for the three-year survey as well. (See Appendix 
Tables B.l and B.2.) 
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Box 1 



Types of Self-Reported Delinquent and Criminal Behavior 

Survey respondents were asked how often they had committed a variety of offenses in the 
last 12 months, with the option of answering “never,” “once,” “twice,” “three or four 
times,” “five or more times,” “don’t know,” or refusing to answer. The types of self- 
reported offenses are listed below. 

Property Incidents 

• Deliberately damaged property 

• Stole something worth more than $50 

• Stole something worth less than $50 

• Entered a house he or she was not supposed to enter 
in order to steal something 

• Drove a car without owner’ s permission 

• Bought, sold, or held stolen property 

• Used someone else’s credit card/bank card without permission 

• Deliberately wrote a bad check 

Violent Incidents 

• Got into a serious physical fight 

• Got into a fight with another group 

• Hurt someone badly enough to need medical attention 

• Used or threatened to use a weapon to get something 

Other Incidents 

• Sold marijuana or other drugs 

SOURCE: Haynie (2001). 



• The program group appeared to be doing less well than the control 
group on some health and lifestyle outcomes. 

Table 8 shows that there are a few statistically significant differences between groups in 
measures of health, sexual activity, and substance abuse. These impacts, which do not neces- 
sarily reflect well on ChalleNGe, were not seen at the earlier survey wave, and there is no clear 
explanation for the findings. 

Most respondents reported that they were in good or excellent health; however, pro- 
gram group members were more likely to be categorized as overweight than control group 
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National Guard Youth ChalleNGe Program 
Table 8 



Impacts on Health, Sexual Activity, and Use of Substances 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Physical and mental health 


BMI Score 2 


25.0 


24.7 


0.3 


0.227 


Overweight 2 (%) 


32.1 


25.9 


6.1 ** 


0.025 


Obese 2 (%) 


11.4 


12.1 


-0.7 


0.726 


Overall health either excellent or good (%) 


69.3 


65.3 


4.0 


0.157 


Psychological distress score b 


5.1 


5.2 


-0.2 


0.552 


Serious psychological distress 13 (%) 


7.1 


8.2 


-1.1 


0.477 


Sexual activity (%) 


Sexually active 


89.6 


89.8 


-0.3 


0.884 


Always uses birth control 


49.8 


57.8 


-8.0 ** 


0.011 


Uses birth control sometimes or most of the time 


33.9 


29.8 


4.2 


0.160 


Never uses birth control 


16.1 


12.4 


3.7 * 


0.096 


Drug and alcohol use (%) 


Binge drinking (5 or more drinks in a row) in the last 14 days 


26.1 


30.2 


-4.1 


0.113 


Frequent marijuana use (10 or more occasions) 


in the last 1 2 months 


26.0 


24.4 


1.6 


0.540 


Ever used other illegal drugs 2 


28.2 


23.2 


5.0 ** 


0.044 


Frequent illegal drug use (6 or more occasions) 


in the last 12 months 


4.7 


4.2 


0.5 


0.705 


Sample size (total = 1.173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Body Mass Index (BMI) is a measure of body fat that applies to both adult men and women. It is 
calculated by dividing a person's weight by his or her height squared. A person is defined as overweight if 
his or her BMI is between 25 and 29.9 and is defined as obese if his or her BMI is 30 or higher. 

b The K6 scale is the sum of the responses to six questions asking how often a respondent experienced 
symptoms of psychological distress. The scale ranges from 0 to 24. A score of 13 points or more on the K6 
scale is considered an indication of serious psychological distress. 

c Other illegal drugs" refers to drugs other than marijuana. Examples given in the survey question 
included: LSD or other psychedelic drugs, cocaine, crystal rneth, amphetamines, barbiturates, heroin, and 
performance-enhancing substances like anabolic steroids. 
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members. Calculated using the Body Mass Index (BMI), 32 percent of the program group was 
overweight compared with 26 percent of the control group. No difference was seen between the 
two groups in those categorized as obese. Looking more closely at the BMI scores, the scores of 
the program group were only slightly higher than the control group and were right at the bor- 
derline between the normal and overweight categories. 

Across the sample, 90 percent of respondents reported that they were sexually active. 
Surprisingly, the program group reported that they were less likely to use birth control than the 
control group. It is possible that some of these sample members, while still relatively young, 
may be actively trying to have children. Additional analysis of the available information on this 
sample could not confirm this hypothesis. Those who reported never using birth control did not 
match closely with those who are married or living with a partner. 

As shown in the bottom panel of Table 8, the survey also asked sample members about 
the frequency of their alcohol and substance use. Rates of recent binge drinking (5 or more 
drinks in a row) and frequent use of marijuana and other illegal drugs are similar across the pro- 
gram and control groups. However, a higher percentage of the program group reported having 
ever tried illegal drugs other than marijuana. 

• Program group members were more likely to be living on their own at 
the time of the survey. 

The survey also measured rates of marriage, parenthood, and living independently, all 
potentially related to the transition to adulthood. As Table 9 illustrates, the program group was 
less likely than the control group to still be living with their parents (45 percent compared with 
50 percent). This is comparable to recent national statistics that found that nearly 50 percent of 
18- to 24-year-olds lived in their parents’ home. 40 Twenty-five percent of the program group 
and 20 percent of the control group were living in their own home at the time of the survey; this 
difference is statistically significant. These figures are approximately 5 percentage points higher 
than in the 21 -month survey for each group, likely illustrating further steps toward indepen- 
dence. A similar proportion of each group also reported being married or living with a partner, 
but only about half of those living on their own were also married or living with a partner (not 
shown in table). 

Just over a quarter of sample members reported having at least one child or being preg- 
nant at the time of the interview. Again, the research team hypothesized a link between mar- 
riage, independent living, and parenthood. However, further analysis indicated that parents or 



40 Rumbaut and Komaie (2007). 
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National Guard Youth ChalleNGe Program 
Table 9 



Impacts on Living, Marital, and Parental Status 



Outcome (%) 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Living and marital status 


Living at parents' home 


45.1 


50.8 


-5.7 * 


0.055 


Living in own home or apartment 


25.0 


20.0 


5.1 ** 


0.041 


Living in someone else's home 


22.1 


21.1 


1.0 


0.691 


Other living arrangement 3 


7.8 


8.1 


-0.4 


0.828 


Never married and not living with partner 


74.2 


78.0 


-3.9 


0.126 


Married or living with a partner 


24.2 


20.4 


3.7 


0.131 


Divorced or separated 


1.7 


1.5 


0.1 


0.869 


Parental status 


Has at least one child or is pregnant 


28.3 


25.1 


3.2 


0.214 


Has a child 


27.9 


24.5 


3.5 


0.180 


Pregnant 


0.9 


1.4 


-0.5 


0.357 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. 
Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 
Rounding may cause slight discrepancies in sums and differences. 

a "Other living arrangement" includes: living in a group home or homeless shelter, not living 
anywhere (homeless), or any other arrangement not specifically listed. 



pregnant sample members were not more likely to be living either on their own or with a 
spouse. Given that a majority of sample members were male, many fathers were likely not liv- 
ing with their children. 



Results for Subgroups 

Some programs are more or less effective for specific subsets of the population they 
serve. Data on such subgroup effects can inform decisions about how to target scarce program 
resources. Analyses examined whether the impacts of ChalleNGe differ for four sets of sub- 
groups defined by characteristics measured at the point young people entered the study: age, 
high school academic performance, whether a family member suggested ChalleNGe participa- 
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tion (a possible sign of family involvement), and previous involvement with the justice system. 
Tables 10-13 show key outcomes for these subgroups. 

During field visits, ChalleNGe staff reported that younger participants can be more dif- 
ficult to work with because they are less focused and mature. Moreover, participants who are 16 
or 17 years old when they finish the Residential Phase tend to have fewer options for postresi- 
dential placements. Table 10 shows selected impacts for sample members who were 16 years 
old at the time of enrollment and for those who were 17 and 18. 41 The last column of the table 
(labeled “Difference between subgroup impacts”) shows whether the differences in impacts be- 
tween the two subgroups are statistically significant. There is some evidence — supporting 
ChalleNGe staff suppositions — that the program has a more positive effect on older partici- 
pants. For example, program group members in this older age group were more likely than the 
control group to have earned a high school diploma or a GED and to be involved in another 
productive activity, such as attending college, working, or serving in the military. ChalleNGe 
appears to have reduced high school diploma receipt among the subgroup of younger sample 
members; some of them probably obtained a GED through the program when they otherwise 
would have obtained a regular diploma. 

Tables 11-13 show no clear patterns of differences in impacts for the remaining 
subgroups. 



Site Comparisons 

Questions often arise about the differences across sites in the impact analysis. As dis- 
cussed above and in greater detail in earlier reports on ChalleNGe, while the core components 
of ChalleNGe were consistent across sites, implementation did vary based on affiliation, loca- 
tion, and staffing of the individual sites. 42 The implementation research did not systematically 
catalog the site differences, so the qualitative information needed to understand differences 
across sites is limited. Quantitatively, sample sizes in many of the 10 sites are too small to an- 
alyze the impacts at the site level. 43 While impacts do appear to vary slightly by site, they do not 
vary significantly or in a way that suggests that the impacts of one site are driving the results. 



41 As noted earlier, youth who were under 16.5 years old at the time of enrollment were not included in the 
study. 

42 Bloom, Gardenhire-Crooks and Mandsager (2009). 

43 See Appendix Table A.l for sample sizes by site. 
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National Guard Youth ChalleNGe Program 
Table 10 

Selected Impacts, by Age at Random Assignment 
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of variance (ANOVA) tests are interpreted. 




National Guard Youth ChalleNGe Program 
Table 11 

Selected Impacts, by Delinquency Level at Random Assignment 

Delinquency Level 
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National Guard Youth ChalleNGe Program 
Table 12 

Selected Impacts, by Academic Performance at Random Assignment 

Academic Performance 
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National Guard Youth ChalleNGe Program 
Table 13 





36 




' o 
5 & C 

l c O 

? -S’® 

C ft 
> o g 
. as 

\ ~0 03 C£ 

D P 03 



i § « , 

i .S ?? c3 
, ,, *3 ft 



r a u a 

; A s-h O 
i O 3 >n 
, <C “ T 3 

i > 2 a 

: o a g 



<U <U S 

y 3 m 

o c 

a « <u e 

,“ra s;l 



37 




In-Depth Interviews with Challenge Graduates 



Most young adults are uncertain about the future. Many, even those who go on to four- 
year colleges, take some time to make progress toward their long-term goals. For those who 
struggle through high school, though — failing classes, getting into trouble with teachers, or 
growing up without a supportive or resourceful network of adults — the transition to adulthood 
can be a particularly vulnerable time. Although each participant’s story is unique, these are 
largely the youth for whom ChalleNGe was designed. As was shown in Table 2, nearly 80 per- 
cent of the program’s participants listed “getting their life on track” as a reason for their interest 
in ChalleNGe. Most of the participants were receiving poor grades in high school, and a large 
percentage had been suspended from school or arrested. In order to better understand how well 
ChalleNGe succeeded in helping these young people, MDRC conducted in-depth interviews 
with 24 graduates of the program. 

Overall, the interviews — like the program’s impacts — tell a mixed story. Nearly all 
the interviewees described a striking transformation in their attitudes and self-confidence as a 
result of their participation in ChalleNGe. The program made them more aware of their own 
potential and cultivated a sense of worthiness and agency. High school had not been working 
for them, and although many did not like ChalleNGe at first, all of them expressed apprecia- 
tion for the combination of the strict regimen and individualized support they experienced in 
the program. 

However, most also said that they had struggled to maintain progress after they left 
ChalleNGe. While most had earned their GEDs in the program and landed a job shortly after, 
by the time of the interviews, few interviewees had made much educational progress, and most 
had had two or more low-skilled, low-wage jobs. Despite their fond memories of the program 
and their strengthened resolve to improve themselves, the graduates who were interviewed felt 
that they had lost momentum since they had graduated from the Residential Phase. 

This chapter presents some of the key interview findings, as well as common themes 
and challenges that interviewees discussed. It is important to keep in mind that the young people 
who were interviewed had returned home to one of the worst labor markets in U.S history. Var- 
ious studies have shown that young adults and low- skilled workers have been among the hard- 
est hit by the economic recession. The fact that most of the ChalleNGe graduates had, at best, 
either a high school diploma or a GED, meant that they were at a particular disadvantage com- 
pared with those who had more employment experience or educational credentials. 44 



44 Sum, Khatiwada, and McLoughlin (2009); Sum, Khatiwada, and Palma (2010). 
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Methodology 

Throughout August and September 2009, two MDRC researchers conducted 24 in- 
depth phone interviews with ChalleNGe sample members who had been randomly assigned to 
the program group, graduated from the ChalleNGe program, and completed the three-year fol- 
low-up survey. 

Westat, Inc., who administered each follow-up survey in this study, contacted a pool of 
program group graduates who had recently completed the three-year survey, and asked them if 
they would participate in another, less-structured interview. Westat provided MDRC with the 
names and contact information for 96 sample members who expressed interest in being inter- 
viewed. The MDRC researchers contacted the pool of 96 sample members, making call at- 
tempts to participants in each of the ChalleNGe program states in the evaluation. Once 24 inter- 
views were conducted, researchers stopped seeking interviews. Since all the interviewees had 
previously agreed to be interviewed, all were amenable and interested when they were con- 
tacted by MDRC. 

The interviews averaged 30 minutes in length and included questions about the 
graduates’ current living, work, and school situations; their experience with the ChalleNGe pro- 
gram; and what their transition was like after ChalleNGe. Generally, interviewees were easy to 
reach. Most were contacted after one to three call attempts. Individuals who completed the in- 
terview were sent a $50 money order to compensate them for their time. 

Of the 24 completed interviews, 16 were male and 8 were female. It is notable that fe- 
males were overrepresented in the in-depth interviews. Graduates from 9 of the 10 ChalleNGe 
programs in the study were represented among the 24 completed interviews. 

Interviewees entered the ChalleNGe program between the ages of 16 and 18 years old. 
Most of the 24 interviewees had graduated from the Residential Phase in December 2006 and 
had completed the one-year Postresidential Phase by the end of 2007. Interviewees were con- 
tacted approximately two years after they had completed the Postresidential Phase and were on 
average 20 years old at the time of the in-depth interviews. 

It is important to note that the individuals who participated in the in-depth interviews 
were not representative of the program group as a whole but rather of a subset of the most moti- 
vated sample members. The 24 individuals who were interviewed were motivated to graduate 
from the ChalleNGe program. They were amenable to completing the three-year survey as well 
as participating in the in-depth interview and were easy for interviewers to reach. However, the 
interviews reveal that even this particular group of ChalleNGe graduates found it difficult to 
maintain momentum and find support after completing the program. 
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Road to the ChalleNGe Program 

The 24 ChalleNGe graduates who were interviewed told of a range of experiences in 
their lives before and after participating in the program. 

Most interviewees had first learned about the program from their parents, who them- 
selves heard about it from a school administrator, television, or the local newspaper. Some in- 
terviewees reported having a family member or a friend who had graduated from ChalleNGe. 
Most said that their families were supportive and relieved that they had made the choice to par- 
ticipate in the program. In contrast, their friends were often unsupportive, primarily because the 
program would take them away from their friends and lifestyle. 

The graduates described a range of reasons for being drawn to ChalleNGe: They felt 
disengaged from school (truancy, disruptive behavior, disrespect for teachers and authority, a 
sense that teachers did not care about them); they had conflicts with their parents, were in nega- 
tive social environments, had a history of substance abuse, desired to have a career in the mili- 
tary, or wanted to get a GED or a high school diploma, even though they were behind in school. 
What united them was that they were having trouble in high school and the traditional high 
school path was not working for them. 

While many interviewees reported that their transition to the residential portion of Chal- 
leNGe was not easy, almost everyone said that they were glad they had completed the program. 
Some stated that they wished they could go for a second time. “I’m extremely happy I did the 
program,” said one graduate. “When I’m in a situation now, I put myself back where I was three 
years ago... in ChalleNGe, and I can apply what I learned there.” Another interviewee had a 
similar view: “I think it's the best thing my mom ever made me do. I’d go back and do it again. I 
learned how to value myself more... the ChalleNGe program made me a lot stronger so I was 
able to deal [with life] when I got back. I think [the ChalleNGe program] made me grow up.” 

Many of the interviewees also expressed particular appreciation for the structure and 
discipline of the program. While not all of them had positive feelings about it right away, many 
strongly emphasized that this aspect of ChalleNGe was what they ended up liking the most. “I 
loved it,” said one graduate. “I love discipline and hard stuff. I love stuff that builds me up and 
makes me a man.” 

The majority of the 24 interviewees spoke highly about the residential portion of the 
ChalleNGe program. Their struggles to maintain the momentum they found while in the pro- 
gram varied and are discussed further in the sections that follow. 
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ChalleNGe Interviewees’ Current Status: Employment, Education, 
Living Situations, and Social Lives 

This section focuses on ChalleNGe interviewees’ experiences with work, school, and 
living and social situations upon graduating from the program. Table 14, referenced throughout 
this chapter, summarizes interviewees’ current employment and education status and where 
they were living at the time of the in-depth interviews. These ChalleNGe graduates encountered 
typical roadblocks that many young people face as they take steps toward independence and 
adulthood. Naturally, some of the graduates had more success than others in school and work 
and developing independence. While each interviewee had unique experiences, similar themes 
emerged as they recounted their experiences in returning home from the ChalleNGe program. 

Employment 

As the three-year survey results show, ChalleNGe participants were more likely to be 
working compared with control group members three years after the Residential Phase. The 
interviews with the 24 program graduates offered a fuller picture of some of the jobs partici- 
pants had found after ChalleNGe. 

Every interviewee had held at least one job since graduating from the program. In most 
cases, the graduates who were interviewed had found work quickly after graduating. Some of 
them had set goals for themselves to get a job within two weeks of leaving the program, often in 
fast food chains, retail stores, or manual labor jobs. They had varying success holding these 
jobs, but only a few had remained in the same position for more than six months. More often, 
interviewees had held a handful of positions, with short periods of no work. At the time of the 
interviews, several graduates had been recently laid off and identified the economic recession as 
a major barrier to finding new work. 

The experience of finding the initial job, regardless of how long they had kept it, ap- 
peared to have had a sustained effect on graduates’ desire to work. One interviewee had 
moved from one part-time job to another since returning from ChalleNGe. At the time of the 
interview, she was working part time as a waitress. This young woman was confident that she 
would find a “better” job soon. She had the realistic perspective that she needed a job that 
paid her bills, but also wanted to find one that was more permanent and career oriented. She 
elaborated: “You need something to fall back on, especially with this economy. I really want 
to be able to have options.” 

As noted, every interviewee had found work relatively quickly upon returning home, 
usually in at fast food chains or retail stores, or in manual labor. These jobs are consistent with 
what one would expect a 19- or 20-year-old with little work experience to find. Additionally, 
since ChalleNGe was not a vocational training program, graduates did not necessarily have a 
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National Guard Youth ChalleNGe Program 
Table 14 



Current Activity, Among In-Depth Interviewees 



Outcome 


Number of Interviewees 


Employment 




Working or in the military 3 


13 


Not working 


11 


Education 




In school 


9 


GED certification class 


2 


Community college degree or certificate program 


7 


Not in school 


15 


Living arrangement 




Living with parents or another family member 


14 


Living with friends or roommates 


4 


Living with spouse or boyfriend/girlfriend 


4 


Living alone 


1 


Living on army base 


1 


Sample size 


24 



SOURCE: MDRC in-depth interviews, 2010. 

NOTE: “Interviewees reported working in the following fields: construction and warehouse work, food service, 
direct care, and daily temp work. Three interviewees were enlisted in the military. 

specific trade they could apply in their job search. Many interviewees did not hold on to their 
jobs for long and expressed a desire to get a “better” or higher-paying job or to pursue an educa- 
tional goal. Two interviewees were working and in school at the same time. This pool of pro- 
gram graduates was focused on earning money they could save, often for school or to move out 
of their current living situation. 

As Table 14 shows, at the time of the interview, more than half of the interviewees were 
working or in the military. This does not include those interviewees who claimed to be looking 
for work. Those with jobs worked as cashiers in supermarkets and fast food restaurants or 
waited tables. Others held jobs in fields related to their career interests: teacher's aides in day- 
care centers or home health aides in nursing homes. Most of those who were currently without a 
job were optimistic that they would find work. One interviewee, who had found work right after 
he returned from ChalleNGe but was not employed in his desired field, was confident that he 
was on the right path: “Everything I do, I do as a step to get where I’m going.” 
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Education 

Impact findings also reveal positive effects on participants earning their GED certifi- 
cates. As Table 4 shows, more than 70 percent of ChalleNGe participants earned a GED or a 
high school diploma, compared with 55 percent of the control group. This is because many 
graduates earned their GED certificate while they were still in the Residential Phase. Interviews 
with graduates offered a fuller picture of the kinds of educational programs they were interested 
in and how they had attempted to pursue them when they returned home. 

Most interviewees had completed tenth or eleventh grade before dropping out of high 
school and joining ChalleNGe. Similar to the full program sample, almost all interviewees ob- 
tained their GED certificate while they were in the ChalleNGe program. Those who did not earn 
their GED during the program obtained it shortly after the Residential Phase. 

Table 14 shows that slightly over a quarter of the interviewees were enrolled full or part 
time in a technical certificate program or a community college at the time of the interview. The 
types of programs they were enrolled in included education, criminal justice, psychology, or 
training programs (nursing or EMT). Two interviewees were enrolled in school and working at 
the same time. In many cases, those who were not in school expressed interest in obtaining trade 
certification in a range of fields that included mechanic, EMT, and business. 

Unfortunately, many interviews with ChalleNGe graduates showed that they had made 
little sustained progress toward their educational goals in the three years after they had com- 
pleted the program. At the same time, many interviewees identified continued schooling as a 
key goal and challenge for the years ahead. While those who did not complete their GED in the 
program successfully got their certificate after graduating from ChalleNGe, most of those inter- 
viewed had made, at best, limited progress toward a college degree or a vocational/training cer- 
tificate. Several graduates mentioned that they could not pay for college classes, which resulted 
in a loss of momentum to carry them from one semester to the next. Others cited a change of 
plans that led them to abandon the educational program in which they were enrolled. 

One interviewee described a pattern of starting and stopping school for a few academic 
semesters because he was financing his education with his own income: “Sometimes, I don't 
have money [for the classes], and then, when I have money again, I go back.” Although he did 
plan on going back to school when he could pay for classes, he did not know what he wanted 
to major in. 

As shown in Table 2, fewer than one- t hird of the sample reported that their household 
received any public assistance, indicating that the participants in ChalleNGe, in general, do not 
come from extremely low-income households. While the ChalleNGe population overall is not 
extremely poor, it is noteworthy that none of the interviewees indicated that they received finan- 
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cial assistance from their parents, although the majority were still offered the financial benefit of 
living at home. Many of those who were in school had to pay for their classes on their own. 

Other research reveals that it is common for young people to start and stop their educa- 
tion due to competing demands, particularly for those who must pay for their own education. 
Research by MDRC and others that specifically targeted community college students suggests 
that many students want to earn a degree, but are hindered by the competing demands of work, 
family, and school. Institutional barriers, such as poorly tailored instruction, insufficient finan- 
cial aid, or inadequate student services, can also impede students’ academic progress. 45 

In spite of these discouraging educational outcomes, nearly every interviewee expressed 
a desire to enter some educational program in the years ahead and felt that it was a real possi- 
bility. Many said that they were actively saving money to do so. “I am going to go to school,” 
said one participant. “There is no doubt about it. Hopefully, by the end of this year, I’ll be 
enrolled in school.” This particular participant was working full time to “save up money” so she 
could eventually study cosmetology. Overall, interviewees exhibited a sense of confidence that 
with persistence and hard work, their educational goals could be attained. 

Living Situations and Social Lives 

Two years after completing the Postresidential Phase of the program, interviewees were 
trying to figure out how they were going to approach work and school while developing their 
independence. Most had not saved enough money from their jobs to be able to afford living on 
their own. Table 14 illustrates that at the time of the interview, a little over half of the inter- 
viewees were living with their parents or another family member. Others were living with 
friends or a significant other. One interviewee was living on her own, and another was living on 
an army base. As noted earlier, most respondents were 19 or 20 years old at the time of the in- 
terviews. It is not uncommon for young people, especially those who have not moved on to tra- 
ditional four-year colleges, to still live with their parents. Almost all the interviewees had 
moved back in with their parents when they returned from ChalleNGe. Among those who still 
lived with their parents, many said that they wanted to live on their own at some point in the 
future, but were saving money by living at home. 

Most of the interviewees described having a mix of friends from before and after Chal- 
leNGe, although the interviews revealed a certain tension between their social lives and their 
desire to stick to the goals they had set in the program. Many interviewees did not seem to have 
a supportive social circle to which they could return. While some spoke of returning to their 
communities as role models to their friends, for many others, adhering to their goals appeared to 



45 Brock and LeBlanc (2005); Scrivener and Weiss (2009). 
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demand a high level of social discipline. One graduate who was focused on finding a job de- 
scribed how he spent his days filling out online applications and walking around his town look- 
ing for hiring signs. After this, he said, “I just go back home, watch a movie or something, wake 
up, do it again.” 

Those who felt they had made the greatest progress after ChalleNGe repeatedly de- 
scribed to the interviewers how they had made a conscious effort to avoid the kinds of social 
interactions that had gotten them into trouble before, such as “hanging out on the street,” and 
drinking. “After ChalleNGe, I kinda stayed away from my old friends,” one graduate said. “I 
was afraid to get into a fight because I knew that if I did, I’d just fall right back into my old ha- 
bits.” Another interviewee said that he avoided his old friends altogether and maintained dis- 
ciple by opting not to socialize and, instead, to look for work or stay at home, “to stay out of 
trouble.” 



Other interviewees said that they had been unsure what life would be like with their old 
friends after they returned home from ChalleNGe and felt they had to find out for themselves 
before they made the decision to remain close. One interviewee described how he went back to 
his old habits and group of friends, only to find that he was drinking and “causing trouble to my 
mom.” Within a few more weeks, he “looked in the mirror, let the friends I was running around 
with go, and started in on my life.” 

Those interviewees who did associate with their old friends conveyed a sense that they 
were maintaining stronger boundaries in their social interactions, that they knew how to “do 
their own thing” and would not get pulled back into the behavior that had gotten them into 
trouble before. One graduate described how he was able to hang out with his old friends, while 
maintaining a certain degree of separation: 

I’m in a different place. Now that I’ve gone to ChalleNGe, I’m in a different 
predicament. I don’t smoke weed anymore, but my friends still smoke. It’s 
not a big deal to me anymore. When I first came home [from ChalleNGe] my 
friends were like: “You are different. You are always preaching now.” But I 
try not to preach to them. 

These ChalleNGe graduates described a fragile balance between maintaining old 
friendships while trying to develop new paths and goals for themselves. For some, reaching 
these goals meant that they had to cut ties with friends to avoid bad habits. Others were able 
to socialize without falling back into old patterns. Still others choose isolation as a way to 
avoid old routines. 

Overall, most respondents agreed that they were not as far along as they would have 
liked in meeting their education and employment goals and fulfilling their desire to live on their 
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own. In spite of this inconsistent progress, most did not report falling back to extremes, such as 
drinking and doing excessive drugs, getting arrested, or hanging out with the wrong people. Box 
2, Isaac’s Story, on p. 48, provides an illustration of the challenges that many interviewees faced 
upon returning home from the program. 

The next section will discuss how the ChalleNGe program affected interviewees’ per- 
ceptions of themselves, their outlook on life, and their attitudes toward the people around them. 



Challenge’s Effects on Self-Image and Attitude 

Beyond the effect that ChalleNGe had on its graduates’ employment and education, it 
also profoundly affected their attitudes about themselves and their progress to adulthood. When 
asked to reflect on the program’s enduring effect on their outlook, the interviewees consistently 
described a maturation that one would not expect in a five-month period. Of course, as has been 
noted, those who were interviewed are not representative of all program participants. It may not 
be surprising that these particular sample members would describe learning positive and endur- 
ing lessons during their time in the program. But, nonetheless, it is noteworthy that their 
descriptions of ChalleNGe’ s effect on their self-understanding and outlook tended to cluster 
around a small group of themes: confidence and responsibility, feeling of self-control, and sense 
of leadership and potential. 

Even two years after completing the program’s Residential Phase, the graduates who 
were interviewed pointed to the experience as a turning point in their understanding of what 
steps they would need to take to move toward their goals. In spite of the continuing difficulties 
many of them faced, they spoke with an understanding that they were responsible for their own 
progress and that they would have to take initiative to move in a positive direction. 

One graduate, a young man who was training to be a firefighter, summed it up this way: 
“To achieve a goal, you have to put your mind to it. ChalleNGe taught me how to really do that, 
how to see that you really have to achieve a lot of small stuff to get to the big stuff, so that 
[when you get out], you can actually have a thought process about what you need to do to deal 
with a situation.” 

Another, a young woman who was preparing to start community college classes for a 
career in social work, said: “They prepared us to make it through the first steps and then make it 
on our own.” While graduates faced continued difficulties meeting their goals, this sense of in- 
vestment in the future may be responsible — at least in part — for the enduring impacts on 
idleness that are shown in Table 6. 
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Box 2 



Isaac’s Story 

Isaac* graduated from the California ChalleNGe program in December 2006. He entered 
the program after he fell behind in credits in high school and the principal told him that he 
had the choice of either entering a high school continuation program or attending Chal- 
leNGe. Although Isaac earned his General Educational Development certificate (GED) 
while in the program, he decided to reenroll in high school after graduating from Chal- 
leNGe. But after struggling again, he decided he would be “better off with a good GED 
score than a high school diploma but a low GPA.” 

Isaac loved his ChalleNGe experience. “At Grizzly, there are no class clowns... it’s a 
good educational environment,” he said. “There’s no way to mess around, y’know; they 
help you. Everybody pays attention.” This was a stark contrast to his high school, where 
“the people that don’t want to learn make it hard for everybody else.” 

Upon graduating from ChalleNGe, Isaac set himself the goal of earning his high school 
diploma and getting a job. “I gave myself two weeks [to get a job] and I beat it — I got 
one in a week and a half.” His first job was working at a local taco restaurant, where he 
stayed for about 10 and a half months. When he turned 18, he quit that job and went to 
work at a Walgreen’ s distribution center, where he stayed for about two years until he was 
laid off in March 2009. Since then, he had done occasional work through a temp agency 
but could not find permanent employment, he said, “because of the economy.” 

At the time of his interview, Isaac was still looking for work in the same kind of ware- 
house distribution environment. He was also considering applying to a two-year college 
to improve his job opportunities and was interested in pursuing either criminal justice or 
mechanics. “I want to learn anything that’ll help me find a job, and I already know how to 
do quite a bit on cars,” he said. Isaac picked up his mechanical skills from his father, who 
used to own his own body shop. 

Isaac was living at home with his mother, father, and two younger brothers (ages 13 and 
16) and said he has a good relationship with his parents: “we’re cool.” The motivation to 
help the family is strong for him: His two-week job goal was motivated by the desire to 
not put any pressure on his parents for support, and his long-term goals included the wish 
“to be on my own and still be able to help my parents. That’s one of the reasons I’m look- 
ing for a job — to help them out. Once I see that they’re good, I’ll move back out on my 
own.” He said “back” on his own because he had lived in an apartment with a roommate 
for about nine months while working at Walgreen’ s, but moved back into the house when 
his parents told him they could use some help looking after Isaac’s two younger brothers. 
Isaac’s father recently lost his job at an RV dealership and at the time of the interview was 
collecting unemployment, putting extra strain on the family. 

*This is not the participant’ s real name. 



(continued) 
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Box 2 (continued) 



As far as his daily schedule, Isaac said he tried to get up by 8:00 a.m. most days and 
spent a few hours outside looking for work. After that, around midday, he came home 
and tried to spend time with his family and help out around the house. During the week, 
he did not go out with his friends too much, since he was trying to keep himself from 
falling back into old habits. The friends he was seeing on the weekends were mostly 
people he had met since graduating from ChalleNGe, and although “obviously, we ain’t 
doing nothing right now,” he said they all shared the same goals: “We all want to go to 
college and do something with our lives.” 

Isaac said he was particularly interested in helping his younger brothers stay out of 
trouble. When asked what kinds of activities he would like to integrate into his life, he 
said he was trying to get his brothers to come work out with him or really “anything to 
motivate them to stay out of trouble, too.” He was no longer in contact with his mentor 
from ChalleNGe — a middle-school teacher with whom he had developed a strong rela- 
tionship in school — but he said his parents and his older sister tried to play the role a 
bit. His sister, 22, had just been released from prison after serving two and a half years. 
She’s “been through a lot, but she’s trying to do good. She just realized [she was on the 
wrong path] too late.” 

“Basically,” Isaac said, reflecting on his current progress and goals, “I’m just trying to 
get independent.” He said it had been tough at times since leaving ChalleNGe. “When 
you’re there, you’re in a good environment and you set goals. Then when you get out, 
it’s har'd, it’s different.” He has stayed out trouble since leaving, but the reasons for his 
participation were more academic than behavioral. And although he no longer saw any 
of the staff members at ChalleNGe, he was in touch with a few fellow cadets through 
MySpace. Like most of the other graduates who were interviewed, Isaac said: “If I 
could do it again, I would.” 



As the interviewed graduates told it, the structure of the residential period offered some- 
thing of a clean slate. While many struggled to maintain the attitudes and behaviors they had 
learned in the program, their experience in ChalleNGe gave them an opportunity to break bad 
habits, understand the consequences of actions they might have taken impulsively in the past, and 
focus their attention on what they needed to do to succeed in the future. Several of the inter- 
viewees explicitly pointed to the heightened sense of responsibility they gained in ChalleNGe. 

A young man who had gotten into trouble in high school several times for fighting de- 
scribed the way the program continued to influence his thinking: “When I’m in a [potentially 
violent] situation now, I put myself back where I was three years ago. . .and I can apply what I 
learned there. For the first time, I haven’t lashed out at anyone. I’ve learned how to keep a clear 
head and mind.” Another echoed his sentiment: “Going through the academy helped me under- 
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stand that high school is over and the real world is coming. There are going to be people you 
don’t like, or that you want to hit, but you can’t do it.” 

The descriptions that interviewees gave of their enhanced sense of leadership and po- 
tential are interesting. While Appendix Table B.l showed no significant impacts on leadership 
and life-coping measures, the graduates who were interviewed repeatedly spoke about these 
themes without being prompted. 

One young man put it succinctly: “I learned how not to be a follower and how to be a 
leader. I also learned how to focus more in class. I know I can do anything I put my mind to.” 
Another graduate felt similarly: 

It was an experience that I needed — I needed someone to push me. Chal- 
leNGe pushed me to my limit. I never really had anyone that said: “you can 
do this.” That’s what they did. I also got a lot of common sense out of the 
program. I learned how to think for myself instead of following everyone else. 

Despite the lack of impacts shown in Appendix Table B.l, the program appeared to 
have a real effect on these graduates’ belief in their own ability to succeed with enough work. It 
is possible that the program exposed them to a heightened sense of possibility, which could 
make goals such as college and skilled employment seem more real than they otherwise would 
have. 



Maintaining Momentum 

Despite the sustained effect on graduates’ attitudes about themselves, for most of those 
interviewed, the momentum they had gained in ChalleNGe had diminished by the time of their 
interview. As discussed earlier in the report, the core component of the Postresidential Phase was 
mentoring, which was intended to reinforce and sustain lessons from the Residential Phase. At 
the time of the interviews, the formal mentoring component had ended. It is not surprising that 
few interviewees described ongoing interactions with their mentors. However, it is notable that 
few mentioned their mentoring relationships at all when asked to describe the entirety of their 
experience in the ChalleNGe program. Those who did bring up their relationship with a mentor 
described it as positive and supportive, but not as a formal “mentor-mentee” relationship. 

As these ChalleNGe graduates described it, the program was a transformative expe- 
rience in their understanding of themselves and their capabilities. However, the abrupt end of 
services and inconsistent mentoring left them largely reliant on the supports that already existed 
in their lives. For some, this meant very little support. Given their young age and the problems 
they had before they entered ChalleNGe, the interviewees’ struggles to gain a foothold in the 
labor market and make strides in their education are not inconsistent with those of many of their 
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peers. Low-wage jobs and inconsistent engagement in postsecondary education are common 
experiences among young people, particularly those with fewer supports and resources. 

What was most salient from the interviews were the graduates’ descriptions of their atti- 
tudinal development and self-esteem. Many attributed an enhanced sense of leadership and po- 
tential to the support that they received at ChalleNGe — a novel experience for many of them. 
Several described the combination of structure and encouragement as integral aspects of the 
program. Conversely, the absence of both in the communities that many returned to surely con- 
tributed to their difficulties in sustaining the progress they had made in the Residential Phase of 
the program. 

As one graduate put it, “Yeah, the lessons slip. If you’re not in that environment and 
you don’t have someone on your butt, it’s gonna slip.” It fades, he thinks, because “you gotta be 
your own boss.” Another interviewee spoke about the contrast between her life in the Chal- 
leNGe program and her life when she returned home: “They don’t tell you what it’s going to be 
like when you go back to the real world. In ChalleNGe, you know what you are doing every 
day, but when you go back out, you have to deal with money, and parents, and the drama.” 

Incorporating enhanced support systems into the Postresidential Phase could ease this 
transition and, potentially, solidify more of the impacts that came out of the Residential Phase. 
In the final section of this report, some suggestions are offered about ways to strengthen these 
components as the program’ s graduates adjust to life after ChalleNGe. 
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Conclusions and Next Steps 



Three years after entering the study (and one and a half years after the end of the pro- 
gram’s Postresidential Phase), young people who had access to ChalleNGe were doing better 
than their counterparts in the control group in several respects. They were more likely to have a 
General Educational Development certificate (GED), more likely to have at least started col- 
lege, and more likely to be working. The lack of impacts in some other areas, such as crime and 
delinquency, is somewhat surprising. But since the sample members were only 20 years old, on 
average, when they responded to the three-year survey, it is too early to know how their transi- 
tion to adulthood will progress. It is not clear at this point whether the evaluation will be able to 
collect further follow-up data. 

The main goals of the ChalleNGe Residential Phase are to help participants obtain a 
secondary school credential and develop attitudes and behavior patterns that will help them suc- 
ceed in the future as students, workers, and citizens. The evaluation’s qualitative and quantita- 
tive data suggest that the Residential Phase succeeds in this respect for many participants. Yet, 
the data also suggest that it is difficult for many young people to maintain momentum afterward 
in a society and labor market that offer few opportunities for young people who have limited 
family support and do not follow a linear pattern from high school to college. The survey data 
show that several of the program’s key impacts have diminished in size over time, and in the in- 
depth interviews, some program graduates (like many people their age) appeared to be having 
difficulty gaining a firm foothold in college or in the labor market. 

The designers of the ChalleNGe program anticipated this problem and added an innova- 
tive Postresidential Phase built around mentoring to try to ease participants’ transitions to their 
lives at home. However, the evaluation found that the Postresidential Phase is implemented un- 
evenly across sites. Moreover, while mentors may provide vital emotional support to young 
people, they may not have the practical expertise and connections to help them find good jobs 
and succeed in college. 

Thus, the evaluation results to date suggest that the program may want to experiment 
with ways to enhance the Postresidential Phase of the program. Possibilities might include: 

• Stationing some postresidential staff in the areas where concentrations 
of former participants live, rather than at the program site. Staff who 
live and work full time in these key communities could focus on develop- 
ing strong linkages with local institutions like colleges, training programs, 
and employers. They could also plan alumni events and have regular one- 
on-one meetings with program graduates (even beyond the first year after 
graduation) to serve as a resource and help keep the young people on track. 
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Realistically, these staff would only be able to work closely with a fraction 
of graduates, since ChalleNGe participants may come from many areas of 
their states. 

• Using financial incentives to promote ongoing connections between 
former participants and the program, or to encourage success in 
school and/or work. Incentives could be designed to promote positive out- 
comes per se (for example, steady employment or progress in college), or 
they could aim to encourage graduates to stay in touch with program staff 
or mentors, as a means to further these outcomes. Other studies have found 
that carefully designed financial incentives can successfully promote a va- 
riety of positive behaviors . 46 

• Promoting a stronger alumni support network through a combination of 
social media and face-to-face activities such as “booster weekends” at 
the program site. Program graduates could serve as a powerful source of 
peer support and practical advice for one another. Graduates may have a ten- 
dency to drift apart after leaving the Residential Phase, but the program could 
take affirmative steps to encourage them to stay in touch with and support 
one another. Activities for graduates might also include their mentors. 

• Developing stronger connections between ChalleNGe and community 
colleges or other postsecondary institutions in areas where graduates 
live. It is somewhat difficult for ChalleNGe programs to develop linkages 
with these educational institutions because participants come from many 
areas of their states. Thus, it might be necessary to h ir e staff in the key com- 
munities (discussed earlier) or to devote more resources to the site-based staff 
who work with participants to develop their postresidential placements. 
Stronger linkages could help participants make better, more informed choices 
about their postresidential activities. 

• Building a stronger vocational training component, either during or after 
the Residential Phase of the program. “Job Skills” is one of eight core 
components of the ChalleNGe program. However, in many sites, this compo- 
nent is designed to promote career awareness and general employability rather 
than to teach specific occupational skills that might help graduates gain access 
to higher-paying jobs. There are several ways that the program might increase 
its focus on skills training: ChalleNGe might integrate occupational skills into 
the core academic component during the Residential Phase, it might extend 



46 Miller, Riccio, and Smith (2009). 
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this phase or add a second Residential Phase focused on skills-building, or it 
might create linkages with other youth training programs, such as Job Corps 
or Year Up, so that graduates could be channeled into those programs. 

Ideally, such enhancements and others designed by program managers would be rigorously 
evaluated to determine whether they increase the program’s long-term impacts. ChalleNGe is 
an example of a public program that already achieves impressive results, but might be improved 
through a systematic program of innovation and experimentation. 
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Appendix A 

Analyses of Survey Response Bias 




Appendix A assesses the reliability of impact results measured by the three-year survey. 
It also examines whether the impacts for the survey respondents can be generalized to the im- 
pacts for the research sample. The appendix first describes how the survey sample was selected. 
Then it discusses the response rates for the survey sample and for the program and control 
groups. Next, it examines differences between survey respondents and nonrespondents, fol- 
lowed by a comparison of the two research groups among the survey respondents. 

This appendix concludes that the survey is reliable but cautions against generalizing re- 
sults beyond the survey respondent sample. A comparison of the program and control groups 
among the survey respondents shows few differences in background characteristics; some dif- 
ferences are expected due to chance, and those were largely addressed in the analysis model. 
Respondents and nonrespondents differed in some measurable characteristics before random 
assignment. 



Survey Sample Selection 

The research sample includes 3,074 sample members who were randomly assigned 
from June 2005 through December 2006. Approximately half the full research sample, a total of 
1,507 individuals, was selected to be interviewed for the 21- month and three-year survey. This 
sample is referred to as the “fielded sample” (see Box A.l and Table A.l). With a planned anal- 
ysis to estimate a pooled impact where each site is weighed equally, the sample was chosen to 
minimize the variance of estimated impacts in the model. Ideally, this means selecting an equal 
number of program and control group members from each site. However this was not possible 
with the wide variation in site size for ChalleNGe. Therefore, a balanced survey sample was 
chosen from each of the larger sites. In the smaller sites this was not possible; all control group 
members were included but more program group members were chosen. 



Box A.l 

Key Samples 

Research Sample. All individuals in the study randomly assigned from June 2005 
through December 2006. 

Fielded Sample. Sample members in the research sample who were selected for the field 
interview. 

Respondent Sample. Sample members who completed the survey. 

Nonrespondent Sample. Sample members in the fielded sample who were not inter- 
viewed because they were not located, refused to be interviewed, or for other reasons. 
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National Guard Youth ChalleNGe Program 
Appendix Table A.l 
Fielded Survey Sample 



State 


Program Group Control Group Total Sample 


California 


94 


29 


123 


Florida 


95 


62 


157 


Georgia 


94 


94 


188 


Illinois 


95 


95 


190 


Michigan 


95 


94 


189 


Mississippi 


94 


49 


143 


North Carolina 


94 


74 


168 


New Mexico 


66 


26 


92 


Texas 


95 


38 


133 


Wisconsin 


94 


30 


124 


Total 


916 


591 


1,507 



SOURCE: MDRC calculations. 



Survey Response Rates 

Sample members who were interviewed for the survey are referred to as “survey res- 
pondents,” or the “respondent sample,” while sample members among the fielded sample who 
were not interviewed are known as “nonrespondents,” or the “nonrespondents sample.” A total 
of 1,173 sample members, or 78 percent of the fielded sample, completed the survey (79 per- 
cent [total = 722] of program group members and 76 percent [total = 451] of control group 
members). Approximately 89 percent of survey respondents also completed the earlier 21- 
month survey. 

Of the nonrespondent sample, 59 percent (184 out of 31 1) could not be located for the 
interview. 1 Whenever the response rate is lower than 100 percent, nonresponse bias may occur. 
That is, differences may exist between the respondent sample and the larger, fielded sample, ow- 
ing to differences between the sample members who completed a survey and those who did not. 



'Other respondents were not interviewed because they refused, were in jail, in the military, in rehab, 
or deceased. 
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Comparisons Between the Research Groups in the Survey 
Respondent Sample 

Although random assignment research designs minimize the potential bias, there is the 
possibility that the characteristics of the research groups will differ due to the selective nature of 
the survey response process. If this is true, the reliability of impact estimates for the respondent 
sample may be affected. 

Table 2 shows selected baseline characteristics of the program and control group 
members who responded to the three-year survey. Differences emerge among respondents for 
highest grade completed and prior arrest/conviction. For highest grade completed, the charac- 
teristic was included in the analysis model to control for the variation between the two groups. 
For prior arrest/conviction, the program group fared worse, so would likely cause the effects 
of the program to be underestimated, if anything. A logistic regression predicting treatment 
status among respondents confirmed that the control and program groups are similar; baseline 
characteristics cannot predict treatment status (the F-statistic testing joint significance was not 
significant). 



Comparisons Between Respondents and Nonrespondents 
Within the Survey Sample 

In order to examine whether there are systematic differences between those who re- 
sponded to the survey and those who did not, Table A.2 presents an analysis of selected baseline 
characteristics of survey respondents and nonrespondents. 

Some statistically significant differences emerge that limit the ability to generalize the 
results presented in this report beyond the respondent sample. Respondents were less likely to 
have been arrested prior to random assignment. During the survey fielding, the survey firm re- 
ported difficulty in interviewing sample members who were incarcerated and tried to follow up 
and interview them when released, if possible. The respondent group was also more likely to 
give education and training as a reason for applying to ChalleNGe (not shown in table). Con- 
versely, prior to random assignment a significantly higher percentage of the respondent group 
was considered obese, based on their reported height and weight. 
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National Guard Youth ChalleNGe Program 
Appendix Table A.2 



Selected Baseline Characteristics of Survey Respondents and Nonrespondents 



Characteristic (%) 


Respondent 

Group 


Nonrespondent 

Group 


Total 


Male 


88.0 


91.5 


88.8 * 


Average age (years) 


16.7 


16.8 


16.7 


Race/ethnicity 2 


Hispanic 


18.1 


20.7 


18.7 


White 


42.3 


40.3 


41.9 


Black 


33.8 


34.1 


33.9 


Other 


0.6 


0.2 


0.5 


Highest grade completed 


8th grade or lower 


14.5 


13.4 


14.3 


9th grade 


29.9 


36.6 


31.5 


10 th grade 


37.9 


33.5 


36.9 


1 1th grade 


17.0 


15.9 


16.7 


1 2th grade 


0.6 


0.7 


0.7 


Usual grades received in school 
Mostly As and Bs 


3.8 


4.7 


* 

4.0 


Mostly Bs and Cs 


17.0 


22.6 


18.3 


Mostly Cs and Ds 


38.0 


34.0 


37.1 


Mostly Ds and Fs 


41.2 


38.7 


40.6 


Has/had Individual Education Plan (IEP) 


31.3 


31.8 


31.4 


Ever suspended from school 


82.6 


80.2 


82.0 


Ever arrested 


31.6 


39.3 


33.3 *** 


Ever convicted 


18.0 


21.5 


18.8 


Who first suggested you should apply for ChalleNGe? 
Yourself 


26.4 


28.8 


** 

26.9 


A relative 


51.3 


43.9 


49.6 


A school official 


15.4 


16.4 


15.6 


The justice system 


6.9 


10.9 


7.8 


Reasons for applying to ChalleNGe? 


Want a high school diploma/GED certificate 


82.1 


77.8 


81.1 * 


Want to go to college/get more training 


43.5 


38.6 


42.5 


Want to get a job 


40.2 


36.1 


39.3 


Want to join the military 


33.6 


29.5 


32.7 


Want to get life on track 


79.6 


80.8 


79.8 


Sample size 


1,173 


334 


1,507 



(continued) 
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Appendix Table A.2 (continued) 



Characteristic (%) 


Respondent 

Group 


Nonrespondent 

Group 


Total 


Overall health very good or excellent 


66.5 


71.4 


67.6 * 


Taking any medication 


23.0 


22.1 


22.8 


Overweight (BMI 25-29) b 


20.3 


22.5 


20.8 


Obese (BMI 30+) b 


11.1 


7.5 


10.3 * 


Ever drink alcohol or use drugs 


37.5 


39.1 


37.9 


Sample size 


1,173 


334 


1,507 



SOURCE: MDRC calculations using Baseline Information Form (BIF) data. 



NOTES: Missing values are not included in individual variable distributions. 

Rounding may cause slight discrepancies in sums and differences. 

Distributions may not add to 100 percent where categories are not mutually exclusive. 

T-tests were used to assess differences in characteristics across research groups. Statistical significance 
levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

a Race/ethnicity categories were constructed by counting as Hispanic those who checked both Hispanic 
and black or white. None of these sample members are counted as multiracial and grouped in the "other" 
category. 

b Body Mass Index (BMI) is a measure of body fat that applies to both adult men and women. BMI is 
calculated by dividing a person's weight by his or her height squared. A person is defined as overweight if 
his or her BMI is between 25 and 29.9. A person is defined as obese if his or her BMI is 30 or higher. 
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Appendix B 

Life Skills and Civic Engagement Outcomes 




National Guard Youth ChalleNGe Program 
Appendix Table B.l 

Impacts on Leadership and Life-Coping Skills 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P-Value 


Goals and priorities 


Important to graduate from college 


3.5 


3.5 


0.0 


0.926 


Important to have a good job or career 


4.0 


3.9 


0.0 


0.176 


Important to have a good marriage or long-term relationship 


3.4 


3.5 


0.0 


0.393 


Important to stay out of trouble with the law 


4.0 


4.0 


0.0 


0.852 


Good sense of the path I want to take in life and the steps needed 


4.7 


4.6 


0.1 


0.151 


Rarely give up on something I am doing, 
even when things get tough 


4.2 


4.1 


0.1 


0.265 


There is really no way I can solve the problems I have 


1.3 


1.3 


0.0 


0.740 


Support from others 


I am able to ask for help when I need it 


4.7 


4.7 


0.0 


0.278 


I seek advice and support from other people 


4.4 


4.4 


0.0 


0.588 


Group processing skills 3 

Tries to present his/her ideas without 
criticizing others 


3.5 


3.6 


-0.1 


0.127 


Encourages different points of view without 
worrying about agreement 


3.7 


3.6 


0.1 


0.372 


Tries to consider all points of view or possible 
options before forming an opinion or making 
a decision 


4.0 


4.0 


0.0 


0.939 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Unless otherwise indicated, all responses are on a 1-4 scale, with 1 indicating a response of "strongly 
agree" and 4 indicating a response of "strongly disagree". 

a Responses on these items are on a 1-5 scale, with 1 indicating a response of "never" and 5 indicating a 
response of "always." 
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National Guard Youth ChalleNGe Program 
Appendix Table B.2 
Impacts on Civic Engagement 







Average Scale Score 11 




Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Civic engagement 










Collective civic efficacy 


3.31 


3.23 


0.07 ** 


0.030 


Social trust 


2.16 


2.10 


0.06 


0.154 


Social responsibility to be a good person 


4.49 


4.43 


0.07 


0.102 


Belief in civil justice 


4.20 


4.24 


-0.04 


0.315 


Sense of civic competence* 3 


3.17 


3.14 


0.03 


0.415 


Past civic engagement 0 


0.44 


0.43 


0.01 


0.637 


Belief in American promise 


3.32 


3.28 


0.04 


0.568 


Justice-oriented citizen* 1 


3.50 


3.47 


0.04 


0.302 


Conventional citizen scale* 1 


3.51 


3.45 


0.05 * 


0.071 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics and weighted by site size, survey response rates, and program versus control ratios. Statistical 
significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a Except where indicated, table shows average of item responses on a 1 flow) to 5 (high) scale 
b Average of item responses on a 1 (low) to 4 (high) scale. 
c Average of responses to a series of seven Yes (1) or No (0) items. 
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Appendix C 

Unweighted Outcomes 




National Guard Youth ChalleNGe Program 
Appendix Table C.l 

Unweighted Impacts on Education and Training 



Outcome (%) 


Program 

Group 


Control 

Group 


Impact 


P- Value 3 


Earned high school diploma or GED certificate 


71.9 


55.7 


16.2 *** 


0.000 


HS diploma 


32.1 


25.7 


6.4 ** 


0.019 


GED 


56.4 


34.9 


2i 5 *** 


0.000 


Both HS diploma and GED 


16.5 


4.9 


11.6 *** 


0.000 


Earned any college credit 


35.1 


19.6 


15.5 *** 


0.000 


Received college degree 13 


0.8 


0.0 


0.8 * 


0.056 


Ever received vocational training 


41.3 


32.9 


g ^ *** 


0.004 


Received trade license/training certificate 

Current status 

Currently enrolled in 


29.7 


26.9 


2.8 


0.303 


High school or GED prep classes 


7.5 


9.0 


-1.5 


0.356 


College courses 


10.9 


8.7 


2.3 


0.210 


Job training 


4.7 


2.8 


1.9 


0.108 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors 
for all impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 2.775, 
2.939,1.929,2.679. 

b All degrees received are associate's degrees. 
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National Guard Youth ChalleNGe Program 
Appendix Table C.2 

Unweighted Impacts on Employment and Enlistment 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 2 


Emulovment history 

In the last 12 months 


Employed (%) 


88.5 


84.0 


4 6 ** 


0.024 


Earnings ($) 


13,448 


11,064 


2,385 *** 


0.002 


Number of months employed 


8.1 


7.2 


Q 9 *** 


0.000 


Current employment 


Currently working 15 (%) 


58.2 


48.3 


9 9 *** 


0.001 


Current hourly wage under $6 


8.0 


4.1 


3 9 *** 


0.010 


Current hourly wage between $6 and $7.99 


10.8 


12.3 


-1.5 


0.426 


Current hourly wage between $8 and $9.99 


15.7 


14.1 


1.6 


0.473 


Current hourly wage $10 or more 


22.8 


16.7 


6.1 ** 


0.012 


Current average weekly earnings 0 ($) 


241 


194 


47 *** 


0.007 


Currently working full time d (%) 


50.7 


43.6 


7 0 ** 


0.018 


Enlistment history (% ) 


Ever enlisted 


18.6 


16.4 


2.2 


0.320 


Active Duty 


11.9 


9.9 


2.0 


0.276 


Reserves or National Guard 


6.6 


6.1 


0.5 


0.744 


Current enlistment (%) 


Currently enlisted in the military 


8.5 


6.8 


1.7 


0.282 


Active Duty 


4.6 


4.2 


0.4 


0.766 


Reserves or National Guard 


3.9 


2.6 


1.3 


0.216 


Current activity 6 (%) 


Current involved in any productive activity 


64.0 


56.7 


7.4 ** 


0.011 


Has high school diploma or GED certificate and is 


currently involved in a productive activity 


49.4 


36.8 


12.6 *** 


0.000 


Sample size (total = 1.173) 


722 


451 







(continued) 
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Appendix Table C.2 (continued) 

SOURCE: MDRC calculations from responses to the 3-year survey. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Rounding may cause slight discrepancies in sums and differences. 

a Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors for 
all impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 0.267, 2.929. 

b Due to missing wage information for some sample members, percentages in wage categories do not sum to 
total percentage currently working. 

c Weekly earnings averages include zeroes for respondents who were not employed. 

^Working full time refers to working at least 30 hours per week. 

e These measures include any employment, school or GED programs, vocational training, military activities, 
or any residential programs (not listed separately above). 
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National Guard Youth ChalleNGe Program 
Appendix Table C.3 
Unweighted Impacts on Idle Time 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 0 


Any time spent idle 


62.6 


72.5 


-10.0 *** 




Idle for 1-2 months 


14.1 


11.2 


3.0 




Idle for 3- 1 1 months 


36.2 


41.0 


-4.8 




Idle for a year or more 


11.7 


19.8 


_g i *** 




Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 

NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Standard errors are presented for all impacts with a p-value of 0.000. Following are the standard errors for all 
impacts with a p-value of 0.000 (presented in the order in which they appear on the table): 2.832, 2.128. 
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National Guard Youth ChalleNGe Program 
Appendix Table C.4 

Unweighted Impacts on Delinquency and Criminal Activity 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Self-reported arrests/convictions (%) 


Arrested 


49.8 


52.7 


-3.0 


0.318 


Charged with any crime 


36.1 


35.4 


0.7 


0.815 


Convicted of any crime* 1 


26.5 


26.4 


0.1 


0.978 


Convicted of a violent crime 


2.1 


2.8 


-0.7 


0.430 


Convicted of a property crime 


5.4 


4.8 


0.6 


0.652 


Convicted of a drug crime 


7.6 


6.5 


1.1 


0.464 


Convicted of a public order crime 


11.1 


13.3 


-2.2 


0.266 


Self-reported delinquency 11 


Any incidents (%) 


57.0 


54.9 


2.1 


0.480 


Any violent incidents (%) 


48.4 


44.3 


4.1 


0.176 


Any property incidents (%) 


15.9 


18.5 


-2.6 


0.244 


Number of incidents 


1.5 


1.5 


0.0 


0.950 


Number of violent incidents 


0.9 


0.8 


0.0 


0.482 


Number of property incidents 


0.5 


0.5 


0.0 


0.529 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 10 
percent. 

Rounding may cause slight discrepancies in sums and differences. 

Results for arrests and convictions cover the time since random assignment. Self-reported delinquency results 
cover the 12 months prior to the survey interview. 

“Categories of conviction are based on definitions from Langan and Levin (2002). Categories may sum to more 
than the percentage convicted because a person may be convicted of more than one crime during the follow-up 
period. 

b See full list of types of delinquency in Box 1. 
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National Guard Youth ChalleNGe Program 
Appendix Table €.5 



Unweighted Impacts on Health, Sexual Activity, and Use of Substances 



Outcome 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Physical and mental health 


BMI Score 2 


25.0 


24.8 


0.2 


0.352 


Overweight 2 (%) 


32.0 


26.3 


5.7 ** 


0.040 


Obese 2 (%) 


11.5 


12.8 


-1.3 


0.520 


Overall health either excellent or good (%) 


69.6 


64.9 


4.7 * 


0.093 


Psychological distress score b 


5.0 


5.1 


-0.1 


0.815 


Serious psychological distress 13 (%) 


7.1 


7.7 


-0.6 


0.703 


Sexual activity (%) 


Sexually active 


90.0 


89.7 


0.3 


0.857 


Always uses birth control 


50.8 


57.2 


-6.4 ** 


0.043 


Uses birth control sometimes or most of the time 


33.2 


30.2 


2.9 


0.324 


Never uses birth control 


15.9 


12.6 


3.3 


0.141 


Drug and alcohol use (%) 


Binge drinking (5 or more drinks in a row) in the last 14 days 


25.8 


29.2 


-3.4 


0.193 


Frequent marijuana use (10 or more occasions) 


in the last 1 2 months 


25.1 


24.6 


0.6 


0.832 


Ever used other illegal drugs 2 


27.4 


21.6 


5.8 ** 


0.021 


Frequent illegal drug use (6 or more occasions) 


in the last 12 months 


4.5 


3.3 


1.1 


0.343 


Sample size (total = 1.173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * = 
10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

“Body Mass Index (BMI) is a measure of body fat that applies to both adult men and women. It is 
calculated by dividing a person's weight by his or her height squared. A person is defined as overweight if 
his or her BMI is between 25 and 29.9 and defined as obese if his or her BMI is 30 or higher. 

b The K6 scale is the sum of the responses to six questions asking how often a respondent experienced 
symptoms of psychological distress. The scale ranges from 0 to 24. A score of 13 points or more on the K6 
scale is considered an indication of serious psychological distress. 

c Other illegal drugs" refers to drugs other than marijuana. Examples given in the survey question 
included: LSD or other psychedelic drugs, cocaine, crystal rneth, amphetamines, barbiturates, heroin, and 
performance-enhancing substances like anabolic steroids. 
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National Guard Youth ChalleNGe Program 
Appendix Table C.6 



Unweighted Impacts on Living, Marital, and Parental Status 



Outcome (%) 


Program 

Group 


Control 

Group 


Impact 


P- Value 


Living and marital status 


Living at parents' home 


44.8 


49.1 


-4.3 


0.155 


Living in own home or apartment 


25.1 


19.7 


5.4 ** 


0.031 


Living in someone else's home 


22.3 


22.6 


-0.2 


0.927 


Other living arrangement 2 


7.7 


8.7 


-0.9 


0.578 


Never married and not living with partner 


73.7 


78.4 


-4.7 * 


0.065 


Married or living with a partner 


24.7 


20.2 


4.5 * 


0.076 


Divorced or separated 


1.6 


1.4 


0.3 


0.722 


Parental status 


Has at least one child or is pregnant 


28.0 


23.9 


4.2 


0.108 


Has a child 


27.7 


23.1 


4.6 * 


0.077 


Pregnant 


0.8 


1.5 


-0.7 


0.232 


Sample size (total = 1,173) 


722 


451 







SOURCE: MDRC calculations from responses to the 3-year survey. 



NOTES: Estimates are regression-adjusted using ordinary least squares, controlling for sample member 
characteristics. Statistical significance levels are indicated as follows: *** = 1 percent; ** = 5 percent; * 
= 10 percent. 

Rounding may cause slight discrepancies in sums and differences. 

a "Other living arrangement" includes: living in a group home or homeless shelter, not living 
anywhere (homeless), or any other arrangement not specifically listed. 
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About MDRC 



MDRC is a nonprofit, nonpartisan social and education policy research organization dedicated 
to learning what works to improve the well-being of low-income people. Through its research 
and the active communication of its findings, MDRC seeks to enhance the effectiveness of so- 
cial and education policies and programs. 

Founded in 1974 and located in New York City and Oakland, California, MDRC is best known 
for mounting rigorous, large-scale, real-world tests of new and existing policies and programs. 
Its projects are a mix of demonstrations (field tests of promising new program approaches) and 
evaluations of ongoing government and community initiatives. MDRC’s staff bring an unusual 
combination of research and organizational experience to their work, providing expertise on the 
latest in qualitative and quantitative methods and on program design, development, implementa- 
tion, and management. MDRC seeks to learn not just whether a program is effective but also 
how and why the program’s effects occur. In addition, it tries to place each project’s findings in 
the broader context of related research — in order to build knowledge about what works across 
the social and education policy fields. MDRC’s findings, lessons, and best practices are proac- 
tively shared with a broad audience in the policy and practitioner community as well as with the 
general public and the media. 

Over the years, MDRC has brought its unique approach to an ever-growing range of policy 
areas and target populations. Once known primarily for evaluations of state welfare-to-work 
programs, today MDRC is also studying public school reforms, employment programs for ex- 
offenders and people with disabilities, and programs to help low-income students succeed in 
college. MDRC’s projects are organized into five areas: 

• Promoting Family Well-Being and Children’s Development 

• Improving Public Education 

• Raising Academic Achievement and Persistence in College 

• Supporting Low-Wage Workers and Communities 

• Overcoming Barriers to Employment 

Working in almost every state, all of the nation’s largest cities, and Canada and the United 
Kingdom, MDRC conducts its projects in partnership with national, state, and local govern- 
ments, public school systems, community organizations, and numerous private philanthropies. 




